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I. INTRODUCTION 



A. GENEBAI COMMENTS 

The Office of Management and Budget (OMB) Circular No. 
A-76 requires federal agencies to inventory and review all 
cf their Commercial Activities (CA) . A Commercial Activity 
is defined as cne which is managed and operated either by 
the government or by a private firm and which provides a 
product cr service tc a federal agency. Unless the activity 
is an inherently governmental function which is required to 
he retained in-house, it must be subjected to a rigorous 
cost comparison to determine the most economical way to 
perform the work. If the cost study shows that it is cheaper 
for the government tc perform the work, in-house resources 
will he used. However, if it is more economical for the 
private sector to provide the product or services, the func- 
tion is contracted out. 

In 1981, the federal government spent $32.5 billion on 
services cf a commercial nature, including maintenance cf 
equipment, military base operations, and facility support 
such as housekeeping, security, and food services. 

[Ref. 1]. Currently, an estimated U00,000 federal employees 
perform in-house CA functions valued at 520 billion. As 
federal agencies move to comply with A-76 policy, the level 
cf contracting cut of CA functions is expected tc grow 
tremendously. It is imperative that this implementation be 
systematic and well thought out in order to achieve both 
maximum eccncmy and efficiency of operations. [Ref. 2] 



E. STUDY OBJECTIVES 

The principal objectives of this research report are to 
examine the application of statistical quality control to CA 
service contracts and evaluate improved designs fcr contract 
administration organizations. 

C. SCOPE AND ASSUMPTIONS 

Although the Commercial Activities policy is applicable 
to all federal agencies, this study will focus on the 
Department cf Defense (DCD) , with particular emphasis on the 
Navy in general and the Naval Facilities Engineering Command 
(NAY? AC) in particular. It is assumed that the reader is 
familiar with the acquisition process within the Department 
of Defense. 

D. BFSEABCH METHODOLOGY 

An initial literature search was conducted to review 
pertinent instructions, regulations, policy guidance, indus- 
trial literature, and reports applicable to A-76 implementa- 
tion. Interviews and discussions were conducted during 
onsite visits with personnel at the following organizations: 

1. Naval Supply Center, Oakland, CA; 

2. Public Works Center, San Francisco, CA ; 

3. Western Division, Naval Facilities Engineering 
Command, San Eruno, CA ; 

4. Naval Weapons Station, China Lake, CA; 

5. Public Works Center, San Diego, CA; 

6. Civil Engineer Corps Officer School, Pert Huer.eme, 
CA; 

7. McClellan Air Force Base, Sacramento, CA; 

3. Iravis Air Force 3ase, Fairfield, CA; 
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. Public Works Department 



Naval Shipyard Long Beach, 



CA; and 

10. Public Works Department, Naval Postgraduate School, 
Mcnterey, CA. 

Additional telephonic conversations were held with 
personnel in the following headquarters organizations: 

1. Naval Material Command, Wasnington, D. c. ; 

2. Naval Facilities Engineering Command, Alexandria, VA; 

3. Naval Supply Systems Command, Washington, D.C.; 

4. Office of Federal Procurement Policy, Washington, 
D . C . ; and 

5. National Aeicnautics and Space Administration, 
Washington, D. C. 



E. THESIS ORGANIZATION 



Chapter I defines the Commercial Activities policy and 
presents the authors' objectives and methodology. Chapter II 
discusses the scope, requirements, and implementation of 
A-76. Chapter III identifies key issues in CA conversions, 
while Chapter IV examines the application of statistical 
quality control to CA contracts. The next two chapters 
address the planning, budgeting and design cf the service 
contract administration organization. Finally, chapter VII 
presents the study's recommendations 
contract conversions. 



:cr improvements in CA 



1 3 



II. COMMERCIAL ACTIVITIES PO LIC Y 

A. DEVELOPMENT OF TEE POLICY 
1 . Executive B ranch 

The government's policy for not competing with the 
private sector was first established during President 
Eisenhower's administration. In 1955, the Bureau of the 
Eudget (ECB) , the predecessor of the Office of Management 
and Eudget (OMB), published Bulletin Number 55-4 which 
stated: 

It is the aeneral policy of the administration that the 
Federal Government will not start or carry on anv 
commercial activity to orovide a service or product for 
its own use if such a product or service can oe produced 
from private enterprise through ordinary business chan- 
nels.' Exceptions to this policy shall be made by the 
head of acency only where it is clearly demonstrated in 
each case that it is not in the puolic interest to 
procure such products or services from private enter- 
prise. [ Ref. 3] 

In 196 6 the first Circular No. A-76 was issued by 
B05 and it represented a major change in previous policy 
statements concerning contracting out. A-76 reaffirmed "the 
Government's general policy of relying on the private enter- 



prise sy 


St<z 


m cc 


supply its needs" [Ref. 4] 


but 


it also 


recognize 


3 


so me 


instances where "it is in t 


he 


r.at ional 


interest 


f cr 


the 


Government to provide directly 


the 


products 


and servi 


css 




uses." [Ref. 4] 







Ihe basic policy underwent another major change with 
the issuance of CM3 Circular A-76 (revised) dated 29 March 
1979. Unlike the previous statements which only stressed 
government reliance cn private enterprise, the new policy 
has three guiding principles. [Ref. 5] 
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a. Rely on the Private Sector. The Government's busi- 
ness is net tc be in business. Where private sources 
are available, they should be looked to first to provide 
the commercial or industrial goods ana services needed 
by the Government tc act on the public's behalf. 



b. Retain Certain Governmental Functions In-House. 
Certain functions are inherently governmental in nature, 
being sc intimately related to the oublic interest as tc 
mandats performance by Federal employees. 



c. Aim for Economy; Cost Comparisons. When trivets 
performance is feasible and no overriding factors 
feauire in-house performance, the American people 
deserve and expect the most economical performance and, 
therefore, rigorous comparison of contract costs versus 
in-heuse costs shculd be used, when appropriate, tc 
decide hew the work will be done. 



TC 

government 

contractor 



support the new emphasis on the economy of 
in-house performance compared to private 
performance, a Cost Comparison Handbook was 
issued as a supplement to A -76 . One year later, in April 
1930, the C apartment of Defense (DCD) issued its own Cost 
Comparison Handbook (DOD 4100.33-H) tc provide even mere 
detailed instructions tc DOD activities. The intended 

purpose cf these handbooks is to provide uniform procedures 
for conducting the ccst comparisons and to improve their 
accuracy and validity. Seme general around rules include: 
[Ref. 5] 

a) Beth the government and commercial costs must be based 
on the same scope cf work and the same level of 
performance. 

b) Standard cost factors will be based or. 

Comparison Handbook. 

c) Full costs are to be used to the maximum extern: 
tical . 



i 



cost 



C uC 



d) For workloads cf a continuing nature, prepriced or 
renewable options shculd be requested from the con- 
tractor to minimize buy-ins. 

e) Services costing under $100,000 annually shculd be 
contracted out without a ccst study unless the commer- 
cial price is unreasonable. 
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f) The cost comparison will use a rate cf 10 percent, per 
annum as the opportunity cost of investments and of 
the net proceeds from the potential sale cf capital 
assets. 

2. legislative E ranch 

As a result cf the increased emphasis on contracting 
out. Congress has repeatedly expressed its concern about the 
implementation or the A-76 policy. In fiscal year 1973 (FY 
78) , Congress placed a one year moratorium on virtually 
ail contracting cut cf A-76 functions if such a conversion 
would displace a Government employes. [Ref. 6] At the start 
cf FY 83, Congress instituted a six month moratorium on 
certain A-76 functions. Although the most recent ban 
exempted studies on grounds maintenance, refuse collection, 
food services, base transportation, laundry and custodial 
functions, it prohibited all other contracting out studies. 
In addition, it prevented the conversion cf contracts for 
all non-exempt A-76 studies completed but not yet awarded. 
[Ref. 7] 

There are also two Public Laws which guide EOD's 
decision to convert. The overall management of DOD 
personnel resources is contained in DOD’s Appropriation 
Authorization Act, 1S75, excerpts of which are provided in 



Appendix A. 


This Act 


re qui res 


that DOD 


"use the least 


costly form 


of manpower 


that is 


cons ist ent 


with military 



requirements and other needs cf the Department of Defense". 
[Ref. 8] Sections of the DCD Authorization Act of 1381 
providing further restrictions are also included in Appendix 
A. The Act states that functions cannot be contracted cut 
to circumvent any civilian personnel ceiling, or unless the 
Secretary cf Defense provides Congress specific notifica- 
tion, certifications, and reports in a timely manner. 
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[Ref. 9] These two previsions are considered to be permanent 
laws which will remain in effect until changed by subsequent 
legislation. 

E. SCCPE OF A-76 
1 . Exclusi ons 

The provisions of 0M3 Circular A-76 dc net apply to 
tne following categories and situations: [Ref. 5] 

a) For contracting cut of personal services that would 
result in an employer-employee relationship. 

b) For major system acquisitions governed by OMB Circular 
A-1C9. 

c) For Contractor Support Services which include 
consulting services, studies and analysis, ana profes- 
sional and management support services. 

a) Whenever i mpie mentati on would be contrary to law or 
inconsistent with the terms of any treaty cr interna- 
tional agreement. 

a) When the activity is performed outside the United 
States, its territories, cr possessions. 

f) When products and services are obtained frem ether 
federal agencies which are authorized cr required by 
law to furnish them. 

g) In times of declared war or military mobilization. 

2 • Inh erently Gc ver nme n ta l Functions 

In addition, A-76 recognizes that inherently govern- 
mental functions must be performed in-house. These govern- 
mental functions fall into three main categories. The first 
catagcry is the discretionary exercise cf Government 
authority. This includes: [Ref. 5] 
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a) investigations, prosecutions and other judicial func- 
tions; 

b) management of government programs requiring value 
judgements such as directing the national defense; 

c) management and direction of the armed services; 

d) conducting of foreign relations; 

a) selection of program priorities; 

f) direction of federal employees; 

g) regulation of natural resources such as the use of 
space, oceans, and inland waterways; 

h) direction of intelligence and counter-intelligence 
operations; and 

i) regulation of industry and commerce. 



'he second category of an inherently governmental 
function involves monetary transactions and entitlements 
such as benefit programs; tax collection and revenue 
disbursements; control of financial accounts and the admin- 
istration of puclic trusts. The last type of government 
function is the conduct of research and development at test 
facilities and the operation and maintenance support of 
laboratories, test ranges, test aircraft and ships. 

[Bef. 5] 

3. Commercial Activities Subjec t to A -7 6 



If a service activity is not specifically excluded 
from CUB A- 76 as previously discussed in Section 31 of this 
chapter, and is not an inherently governmental function as 
defined in Section B2, then it is classified as a Commercial 
Activity. Attachment A to OMB A-76 provides approximately 
one hundred examples of Commercial Activities for fifteen 
different service categories. Such Commercial Activities 
may be operated and managed either by the government agency 
or a private commercial business. 



18 



4. 



G cv (-rn m snt 0 csrati on of a Com mercial Activit y 

In-house performance cf Commercial Activities cannot 
be justified solely on the basis that an activity supports 
cr involves a classified program, or is part of an agency's 
basic mission, or that there is a possibility of a strike by 
contract employees. Government operation of a Commercial 
Activity can only be authorized under one cf the following 
conditions: [Ref. 5] 

a) No Satisfactory Commercial Source Is Available. 
Government operation is permitted whenever it can be 
documented that either: 

i) There is no private commercial source capable of 
providing the needed service, cr 

ii) That the use of a private source would cause 
unacceptable delay or disruption of an essential 
program. The required documentation must b^ 
detailed in terms of cost, time, and performance 
measures. The disruption must be of a lasting 
nature and not just temporary. 

b) National Defense. 

i) Government operation by military personnel is 
permitted whenever: 

1. The personnel are utilized in or subject to 
deployment in a direct combat or combat 
service support role. 

2. The activity is essential for exclusively 
military training. 

3. The activity is required to provide appro- 
priate work assignments for career progres- 
sion cr a rotation base for overseas or sea 
to shore assignments. 

ii) Government operation of a depot or intermediate 
level maintenance facility may be justified to 
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ensure a ready and controlled source cf tech- 
nical competence and resources necessary to meet 
military contingencies. 

c) Lower Cost. If none of the precaeding conditions can 
be met, government operation of a Commercial Activity 
can only be authorized when a comparative cost 
analysis, performed in accordance with A-76 and the 
Cost Comparison Handbook, shows that in-house opera- 
tion has a lower total cost than if it were obtained 
from a qualified private source. 

C. REQUIREMENTS OF 3-76 

1 • Invent or.y cf Commer cial Activi ty F uncti ons 

C MB Circular A-76 required each agency to compile 
and update annually a complete inventory of all Commercial 



SERVICE 


NUMBER OF 
CA FUNCTIONS 


ANNUAL OPERATING 
COSTS (5ILLI0N 3) 


MAN-YEARS 

(THOUSANDS) 


ARMY 


2,941 


7. 1 


133 


NAVY/MARINE 


6,266 


6. 1 


278 


AIR FC5CE 


5,624 


3.6 


260 


OTHER CCD 


454 


. 2 


18 


TOTAL 


15, 287 


1 O 
1 • 

1 £ 


689 



Figure 2.1 DOD Inventory of CA Functions. 

Activities subject to its provisions. The initial COD 
inventory was completed in August 1930 and is summarized in 
Figure 2.1. [Ref. 1C] 
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2 . Man age ment Re vie w 

DCD Instruction 4100.33 requires that a complete 
review of all CA functions he completed during FY 80 through 
FY 84. Subsequent reviews are required at least once every 
five years. In the event that the circumstances supporting 
the initial approval are not subject to change, then subss- 
quent re views may be waived by the Assistant Secretary of 
Defense (Manpower, Reserve Affairs, and Logistics). 

The purpose of the review is to determine whether 
the present method of performance should be continued or if 
the function should be designated for a cost comparison 
analysis for possible change in method of performance. In 
making this determination, the criterion specified in A-76 
and explained previously in this chapter are used. If the 
function is presently contracted, a rough estimate of the 
cost tc perform the work in-house is prepared. If this 
estimated in-hcuse cost is not less than contract perfor- 
mance by 10 percent of personnel cost and 25 percent of cost 
of ownership of equipment and facilities, then the function 
remains contracted out. However, if the likelihood exists 
that in-hcuse performance would meet the cost differential 
criteria, formal cost comparison analysis must be performed 
tc determine the cheaper method. 

3 . Development of Stateme nt of Work 

The preparation of the Statement of Work (SOW) is 
one of the most critical steps in the effective implementa- 
tion of the A-76 policy. Its design will directly impact 
the nature of the solicitations, the cost comparison process 
and subsequent performance either by in-house personnel or 
by contractor employees. The SOW must establish the govern- 
ment's actual minimum requirements for performing the 
service. These standards are the same regardless of whether 
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4 . and A udit I r-Hous e Es timat e 

The DOD Authorization Act, 1931 requires that 
government in-house estimates he based on the "most effi- 
cient and cost effective organization for per f crma nee” . 
[Ref. 9]. DOD Instruction 4100. 33H directs that : 

[Ref. 12] 

Each agency should ensure that Government operations are 
organized and staffed for the most efficient perfor- 
mance. Tc the extent practicable and in accordance with 
agency manpower and personnel regulations, agencies 
sncula precede re views" un der this circular with internal 
management reviews and reorganizations for accomplishing 
the work mere efficiently, when feasible. 
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The activity is not required to achieve this most efricien* 
organization (MEO) prior to a cost comparison study but it 
must use the MEO as the basis for the government in-house 
esti m at s • 

after the in-house estimate is prepared, based on 
the established Statement of Work and using the most effi- 
cient organization, it is required to be audited. The audit 
is performed by an independent audit agency, normally the 
Naval Area Audit Service, and must be started 120 days prior 
to bid opening. [ Eef. 13] In addition to ensuring proper 
compliance with A-76 and the Cost Comparison Handbook, the 
Audit Service reviews and approves the proposed SOW and the 
MEO. 

5. Solicita tio n a nd Ev a luation of. C ontracto r Prop osals 

Cnee the in-heuse estimate has been approved, firm 
bids or proposals will be solicited. Although competitive 
negotiations on a firm fixed price basis are the preferred 
method of contracting, formal advertising and other pricing 
arrangements may be approved in rare instances. [Hef. 14 ] 
After the contracting officer opens the bids or completes 
negotiations, he determines the lowest acceptable contract 
price cf the responsive and responsible bidders. 

6. Cost Comoariscn 



If the lowest acceptable contract price exceeds the 
total in- house cost, then the performance by the government 
is assumed to be cheaper and the cost comparison process is 
complete. When the contract price is less than the in-house 
estimate, further adjustments are required to evaluate the 
impact cf the two alternatives or. the overall cost of 
government operations. 
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There are two adjustments made to the in-house bid. 
First , the cost of capital for assets that will be used for 
in-hcuse performance must be added. Second, any one time 
cost associated with a new start where the function is not 
presently performed in-house must also be included. 

The contract price must also be adjusted for several 
factors. A cost of capital charge may be added if govern- 
ment assets are required tc assure contractor performance. 
Conversion costs are added to reflect the one time costs 
incurred by the government in shirting operations from 
in-house to contract. When contracting out would result in 
a reduction of the present level of capacity, the additional 
amount of overhead which must be absorbed by the remaining 
activities is added to the contract cost. Finally, the 
potential federal income tax revenue which would be paid by 
the contractor is deducted from his contract price. 

After all adjustments have been made, an existing 
in-hcuse function is not converted unless the projected 
contract price is lower than the government estimate by at 
least 10 percent of the in-house personnel related cost. 
This differential is included to account for the possible 
loss of production, the temporary decrease in efficiency and 
effectiveness, and ether unpredictable risks that result 
from contract conversion. [Ref. 12] 

7. Esc i si on and A po eal Proc ess 

Upon completion of the cost comparison, a recom- 
mendation is made to either award the contract or to perform 
the function in-house. The recommendation, along with the 
cost comparison forms are forwarded to the approving 
authority fer review and approval. Once approved, the 
results of the cost study are announced and the detailed 
analysis is made available to the public. If no significant 
discrepancies are identified or an appeal lodged within 5 
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working days (which may be extended by the contracting 
officer up to 15 days for complex decisions), the 

contracting officer will either award the contract or cancel 
the solicitation. In the event the function is to be 
performed in-house, implementation of the MEO must be initi- 
ated withir. 30 days and be completed within one year. 
[Ref. 15] 

D. IfiELEHENTATION OF A- 76 

Currently an estimated 400,000 federal government 
employees perform commercial activities valued at $20 
billion annually. Cf this amount, only $6 billion are 
eligible for cost studies; the other $14 billion worth are 
exempt from A-76 for reasons of national defense. Although 
progress is accelerating rapidly, to date, only a small 
portion cf the eligible functions have received a cost 
comparison. Ihe Office of Federal Procurement Policy (CFFP) 
estimates that a savings of over 35 billion could be 
achieved ever the next five years if these cost studies were 
completed. [Ref. 2] 

Since 1S79, DOD has saved approximately 3140 million per 
year as a result of CA studies. In addition, an average of 
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[Ref. 2] Data 


compiled in 


January 


1982, 


showed that 


60 


percent of the 


functions 


reviewed 


s h i f t e d 


to contract 


and 


one average co 


sts dropped 



19 percent. These reductions were widely distributed 
however, with two-fifths shewing greater than 30 percent 
savings, another two-fifths having savings between 11 and 25 
percent, while the remainder saved 10 percent or less. 

[Ref. 1] 



In FY 82, the Navy had its best year to date in imple- 
menting the A-76 program. A total of 252 cost studies were 
completed ; more than twice the number for the previous 
three year period. Cf the 5,487 civilian man-years studied, 
2,060 were contracted out resulting in a savings to the 
government exceeding 315.9 million. The remaining functions 
that were retained in-hcuse netted a cost avoidance 
exceeding 317 million. [Ref. 16] 

As cf 1 March 1983, the Navy has identified 53,457 
personnel billets as candidates for C A studies. The total 
percentage under NAVFAC’s f unctional/contractual responsi- 
bility was 54 percent. Although NAVFAC still has the highest 
percentage in the Navy, its share has decreased from FY 82 
when it was as high as 70 percent. NAVFAC took an early 
lead in CA contracting for the Navy by providing standard- 
ized Performance Work Statements to its field activities. 
Cf all the NAVFAC functions studied, the four areas of 
transportation, grounds care, building maintenance, and 
janitorial services comprise 82 percent of the tctal. 
[Ref. 17] 
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III. C ONTR ACT CONVERSION ISSUES 



A. PACKAGING OF CA FUNCTIONS 

In i a pigment ing the A-76 policy there are several impor- 
tant contract conversion issues which must be addressed. One 
issue concerns the different methods in which Commercial 
Activities functions can be reviewed and cost compared. For 
example, a single function from the activity's inventory , 
such as bus services, can be studied individually. 
Alternatively, several functions may be combined into a 

package such as vehicle operations and maintenance. When 
most, if r.ct all, functions are consolidated by an activity, 
a total Ease Operating Support (BOS) package is formed. 

A multi-function approach offers many advantages over a 
single function one. It facilitates implementation of A-76 
because it reduces the number of cost comparisons. As a 
result, it is a very appealing option to these commands who 
are under intense pressure to quickly contract out. Recent 
Navy experience has shown that when smaller functions are 
cost compared under A-76 on a single function basis, they 
are extremely likely to remain in-house. However, when 
these same functions are consolidated into a multi-function 
package they are far more likely to be contracted cut. 
[Bef. 16] 

The nature of service contracting is such that manual 
labor alone is insufficient to ensure satisfactory perfor- 
mance. What is needed is an effective management organiza- 
tion that can get the job done properly. A multi-function 
contract increases this requirement and because of its 
greater dollar value, attracts larger firms that have 
increased management expertise. If the functions are 
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complex or trim 5 constrained it may require the vast 
resources of the larger firm. Consolidation of functions 
can also lead tc greater economies of scale and more effi- 
cient use of personnel and material. 

S 

The entire process of contract administration is much 
easier when dealing with a single large con iractcr ratnsr 
than many small ones. The command does not have to contend 
with many differing company policies, procedures, and 

personnel. There is a higher probability for satisfactory 
rework because whenever a problem is identified, only a 
single point of contact need be reached. In addition, there 
are fewer opportunities for contractor tc contractor 
finger- pointing with multi- function contracts. 

B. SHILL BO SI N ESS CC NSI DEE ATIONS 

Multi-function packaging, while of great value in imple- 
menting CM3 Circular A-76, comes in conflict with another 
national policy; the Small and Disadvantaged Business 
Utilization (SADEU) Program. Public Law S5-507, signed on 
24 October 197 8 , pro vides the legal framework for th« SAD3U 
Program. Section 21 of this law states: [Hef. 18] 

It is the policy cf the United States that small busi- 
ness concerns and small business concerns owned ar. d 
controlled by socially and economically disadvantaaed 
individuals, shall have the maximum practicable opportu- 
nity tc participate in the performance of contracts let 
by any Federal Agency. 

A small business concern is one t.iat is independently 
owned and operated, and qualifies under guidelines estab- 
lished by the Small Business Administr ation (SBA) with 
regard tc number of employees and annual receipts. A small 
disadvantaged business is cne owned and operated by a 
minority (Black, Hispanic, American Indian, etc.). In order 
to ensure fair opportunity to participate in government 
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contracts, certain classes of procurements have beer, set 
aside fcr the exclusive participation of small business or 
have been granted 8(a) set asides for small disadvantaged 
businesses. 

Circular A-76 requires that any contracts which have 
been awarded under authorized set-aside programs will ret be 
reviewed fcr possible in-house performance. It also directs 
that functions previously performed in-house that are 

suitable for a set aside program ne awarded without a cost 
comparison. On the ether hand, A-76 states in-house activi- 
ties in excess of 5100,000 annually will not be set aside 
unless the conversion is justified by a cost analysis. This 
last statement has been interpreted to allow unrestricted 
award of multi-f unct icn contracts even though the i car vidua I 
functions, if awarded separately, would require being set 
aside to small business. 

On 16 March 1 982, Congressman Joseph Addab'oo (E-NY) 
expressed his concern to Secretary of the Navy John Lehman, 
that consolidation of base support services under EOS 
contracts would devastate the Navy's S ADBU Program. 
Congressman Addabbo claimed that many of the functions being 
consolidated were traditionally performed by small business. 
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to subcontract to small or small disadvantaged firms for a 
particular service function." [Ref. 19] The Congressman 
concluded by requesting the Navy stop all consolidations. 

In a follow-up letter dated 23 April 1982, Congressman 
Addabbo chided Secretary Lehman for not sending him a 
substantive reply to his original request. He also charged 
that the Navy had accelerated its efforts to exclude small 
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and minority business from service contracts. As evidence, 
he cited the $6.9 million BOS contract awarded at Naval 
Weapons Center, China Lake, California on 18 April 1982. He 
also identified six other 30S solicitations that were 
recently released by other Naval activities. In closing, 

Congressman Addahbo renewed his request to halt all further 
consolidations of CA functions. [Ref. 20] 

In his response. Secretary Lehman stated, [Ref. 21] 

It is my strona belief that consolidated cor.fractina can 
be a cost effective stratsay often fosterina a 'more 
efficient use of scarce acquisition resources. 'However, 

I also believe that consolidations must be undertaken 
with a keen awareness of the objectives of the small 
business program. 

Secretary Lehman maintained that small business has had 
considerable success in capturing multi- function contracts. 
In the China Lake award, he explained that eignt of the 
fifteen functions, representing 50 percent of the contract's 
value, were subcontracted to small business. Finally the 
Secretary promised that a significant portion of future 
consolidated contracts would be awarded to SAD3U firms 
either in total or in part via subcontracting. 

On 24 May 1982, Secretary Lehman sent a memorandum to 
the Chief of Naval Operations (CNO) outlining the Navy's new 
policy on consolidated service contracting. It makes three 
major points: [Ref. 22] 

a) Consolidation of existing small business contracts 
shall only be considered when there is a reasonable 
expectation that it will result in an award to small 
business. In the event that such an award is not 
accomplished, individual function contracting must be 
reinstated . 

b) Functions which are currently being performed under 
8 (a) centra cts will not be considered for consolidation 
unless consented to by the S3A. 
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c) Contracting out of current in-house CA functions shall 
be done giving the SADBU Program careful consideration 
early in the planning stages. In those instances 
where small business participation is determined not 
to be in the best interest of the Navy, supporting 
justification trust be forwarded to CNO (OP 443) prior 
to solicitation. 



Cne week later, on 1 June 1932, Deputy Secretary cf 
Defense Frank Carlucci promulgated the DOD policy that is 
still in effect today. The policy directed that any func- 
tions currently performed by small business shall net be 
considered for consolidation. In addition, the Carlucci memo 
stated that : [Ref. 23] 

Future solicitations, unless there are overriding 
national security considerations, will be oackaaed sc as 
not to preclude performance by small and small disadvan- 
taged concerns as crim e contractors. 

Although the issue appeared to be resolved, it has again 
surfaced with the Cffice cf Federal Procurement Policy 
proposed revision tc A-76 that is currently being consid- 
ered. This proposal would direct that consideration be given 
tc consolidating CA functions into a single statement of 
work for cost comparison and potential contract. While 
admitting that consolidation may reduce prime contract 
opportunities for SACEU concerns, it only directs a reason- 
able balance be maintained between consolidations and single 
function awards to small business. [ Sef . 15] 

In addition to CFPP's recent support of muiti-f urctron 
packaging, the new Deputy Secretary of Defense, Paul Thayer 
has stated that he intends to change the DOD consolidation 
policy issued by his predessor, Frank Carlucci. Recognizing 
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Approximately 300C Navy CA functions involve fewer than 
10 civilian personnel man-years. Sven when consolidated into 
snail multi-function contracts, many will still be within 
the capability of small businesses. Of the 252 cost studies 
conducted in FY 82, twenty were greater than fifty man-years 
and cf these, only five were over one hundred man-years of 
effort. [Bef. 16] Although 65 percent cf the studies were 
restricted to small business, these set asides resulted in 
fewer contract awards. While 75 percent of the civilian 
man-years in the unrestricted solicitations were contracted 
out, only 37 percent of the small business set asides were 
awarded. [Bef. 25] These results clearly demonstrate that 
the larger unrestricted solicitations are mors competitive 
with the government and lead to grsatsr conversions to 
contract. 

It would appear that the Navy cannot achieve the reason- 
able balance sought by 0??P unless it is allowed greater 
latitude in consolidating CA functions. As additional func- 
tions are subjected to CA cost studies, an increasing 
percentage will be mere complex, critical, and efficient 
since the easier and less efficient functions will already 
be contracted out. It is therefore predictable that indivi- 
dual functions and those restricted to small business will 
have ever less success than at present in winning a contract 
award. Thus a total reliance on single function packaging 
will net only be detrimental to small business but will also 
severely handicap the A-76 aim for economy and productivity. 
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C. QUALITY ASSURANCE OP CA CONTRACTS 



When the federal government purchases items or services 
under a contract, it assumes the responsibility for ensuring 
that such items or services conform to stated contractual 
requirements. Two concurrent processes commence at contract 
award which influence satisfactory quality. 

The contractor establishes a quality control program 
whereby management control of materials or services is exer- 
cised for the purpose of prevention of defects. In govern- 
ment contracts, the contractor assumes the responsibility 



for the execution of the 


quality control process. 


[Ref. 26] 


At the 


sane time , 


a government contract 


inspect non 


organization 


administers 


a quality assurance 


process . 



Quality assurance is a planned and systematic approach of 
observing service performance to provide adequate confidence 
that the items or services conform to established technical 
requirements. The quality assurance process verifies the 
required quality of' delivered items or services prior to 
their acceptance. 

Under current A-76 policy, two key documents are 
prepared during the pre-so lac itation phase that influence 
the fcilcw-cn quality control and assurance processes. These 
are known as a statement of work and quality assurance plan. 
The perform an ce-orien ted statement of work describes the 
minimum nsquirea level of services that will be expected of 
the successful contractor. The quality assurance plan states 
the procedures that will be used to check and verify 
contractor performance. 

Prior *c the 1579 revision of A - 7 6 , solicitations 

frequently psed design specifications which established 
precise measurement s, tolerances, or quality control 

requirements for the contractor. Other detailed information 
was provided when deemed necessary. These specifications 
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were primarily intended to obtain standardization cf deliv- 
ered items or services. In CA service contracting, such 
specifications often proved to be unwieldy and ineffective. 
Standardization was net always a suitable objective for the 
service contract process given a wide diversity in 

contractor skills and the character of operations at 

different Navy activities. The government assumed substan- 
tial risk with this method in that it guaranteed acceptable 
results as long as the specifications were followed. 

On 26 January 1962, an OMB memorandum directed that the 
Office cf Federal Procurement Policy Pamphlet No. 4 be 
designated as Supplement No. 2 to Circular A-76. It further 
required that both work statements and quality assurance 
plans for CA functions be written in accordance with this 
new supplement . [ Bef . 27] The OFPP Pamphlet embraces the 

widespread utilization of performance-oriented work state- 
ments instead cf detailed, exacting design specifications . 
The contractor is clearly and precisely tela what is 
required hut not how he must do it. This allows a contractor 
more flexibility in performing the work. An objectively 
defined end product facilitates the contractor's quality 
control ana the government' s quality assurance effort. The 
quality assurance plan gives the government inspector a 
detailed, written plan which allows him to accurately 
assess the contractor's performance. 

The statement of work design encumoers several review 
steps and processes. First, functional areas subject tc ccst 
study must be completely defined. All resource inputs, verk 
processes, and production outputs required for successful 
performance of the job function must be identified and inte- 
grated. After this, a job analysis process occurs in which 
the structure of the organization is designed, an itemized 
listing cf work elements for the function is enumerated, and 
standards of performance for each work element are set. In 
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addition, resources tc accomplish work input are determined, 
performance indicators are listed, and deviation from accep- 
table standards are specified. [Ref. 28] When the job 
analysis is complete, the essential rudiments of a contrac- 
tual statement of work will be formulated. Functional area 
managers then consult with contract specialists and indus- 
trial engineers to complete the contract package. 

The Air Force and Navy have promulgated regulations 
which implement the precepts of OFPP No. 4. Over the past 
three years, both have issued standardized statements of 
work (SOW's) or performance work statements (PWS's) which 
pertain to specific CA functions such as refuse collection, 
grounds maintenance, or any ether areas listed in A-76 
inventories. A field activity performing A-76 cost studies 
will utilize these PWS's and tailor them to incorporate the 
special local re guir ements. Approximately twenty PWS's have 
been written for Navy Public Works commercial functions. 

Standardized quality assurance plans have also been 
prepared for each statement of work and can be tailored by 
local activities. These state the methods that will be used 
in inspecting all contract requirements. The Naval 

Facilities Engineering Command identifies five methods of 
surveillance: [Ref. 28] 

a) One hundred percent surveillance involves inspection 
of each occurence of contract output. It is expensive, 
time consuming, and not 100 percent reliable. 

b) Planned sampling allows part of the contract output to 
be evaluated. It is subjective, and generally useful 
only when certain items of work are very important. 

c) Random sampling uses statistical techniques tc sample 
a portion of ccntract outputs. Each item has an equal 
chance of being evaluated, eliminating inspection 
bias. 
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d) Unscheduled inspections are impromptu spot checks or 
the contractor's performance. 

e) Validated complaints result when customers express 
dissatisfaction with contractor performance. Although 
complaints may not be used in lieu of the ether 
methods, they can be used to verify results of ether 
inspection methods, and make payment deductions. 

The method cf inspection is tempered by the various 
types and frequencies of work. Repetitive, frequent perfor- 
mance may be best suited for planned and random inspection 
methods while infrequent, critical work items may require 
100 percent surveillance . Once a method is chosen, inspec- 
tion schedules are created for each month of contract 
performance . Evaluation worksheets listing each werk item 
are prepared, inspections are conducted, and results are 
recorded. Good performance is rewarded while poor perfor- 
mance is required to be corrected. 

When a contract scli citation is issued, the statement of 
work and other mandatory (boilerplate) provisions are assem- 
bled and distributed to prospective bidders. The Air Force 
has adopted the practice of distributing its quality assu- 
rance plans (less actual inspection schedules) to each 
bidder in an attempt to alert them of the contract's quality 
assurance standards . One key requirement of this entire 
package is clarity, precise wording, lack cf ambiguity, and 
conciseness. Lefty and technical wording tends tc be 
confusing and must be avoided. 

The authors observed that in several instances, more 
attention is given tc preparation of the statements of work 
than to quality assurance plans. This is understandable, 
since many DOD activities are rushing to comply with the 
A-76 requirement to complete cost studies by FY 84. In spite 
cf this time constraint, it would be counterproductive for 
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an activity to move towards implementation of widesp 
service contracting without a specific and definitiz 
for quality assurance. 

The next chapters discuss initiatives in ef 
surveillance methods and properly structured ins 
organizations. A well designed quality assurance 
will result in the optimum use of scarce ins 
resources and will help ensure adequate performance 
service contractors. 
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IV. STATISTICAL QUALITY ASSURANCE TECHNIQUES FOR A-7 6 

CONTRACTS 



A. I NTRCEUCTION 

During A-76 cost studies, three documents are prepared 
which play a key rcle in the Commercial Activities (CA) 
contract process. The first is a contractual statement of 
work which defines minimum requirements for services ordered 
in the contract. The second, a quainty assurance plan which 
is designed and integrated with the statement of work, 
summarizes those schedules and techniques that will be used 
ty the government to verify contractor performance. A third 
essential document, the cost estimate, reflects the mcst 
effective and efficient government employee performance of 
the function being studied. 

Cr.s prime objective of A-76 is cost effective mission 
performance by federal agencies. This is demonstrated by 
comparing the cost estimate cf government employee perfor- 
mance tc prices offered by competing contractors who parti- 
cipate in A-76 firm bid procedures. The low offerer gains 
the right (usually) tc perform the function. 

This emphasis on cost effective performance cf govern- 
ment operations is reflected in that guidance which 
describes the preparation cf A-76 CA quality assurance 
plans. CCD publications describing the design and implemen- 
tation cf quality assurance programs are : 

a) Air Force Regulation 400-28 of September 1979; 

b) Office of Federal Procurement Policy Pamphlet Number 4 
of Cctcber 1980; and 

c) Naval Facilities Engineering Command Manual MC-327 of 
November 1932. 
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Each of these publications calls for development of 
cost-effective quality assurance programs by each field 
activity tc provide reliable assessment of service contract 
performance. Ccst-ef fecti ve surveillance doss connote an 

inspection system being created at the least possible cost; 
more appropriately, the important facet of such quality 
assurance is to utilize existing or new inspection systems 
and resources in a reliable, dynamic fashion to gain the 
best determination of contractor performance under CA 
contracts. It should have the capability to be easily modi- 
fied when CA contract workloads increase. 

The Naval Facilities Engineering Command asserts the 
importance of properly designed quality assurance programs, 
contrasting old and new inspection plan design philosophies: 
[Ref. 28] 



The Navy’s traditional apDroach to surveillance of 
service contracts, often a hit or miss affair with no 
written plan, has not provided adequate quality assu- 
rance.... The method or surveillance which is claimed 
to be used most frequently is 100 percent inspection. In 
reality, however, such inspection is often less than 
total inspection since it is very costly and not always 
feasible. Further, traditional surveillance methods have 
usually focused on the work process (adherence to speci- 
fied steps and frequencies) rather than on the qualitv 
of contract outputs. The net result does not assure 
quality performance.... NAVFAC's new quality assurance 
approach, Dassd on a written plan, is keyed to perfor- 
mance oriented specifications. It focuses on the quality 
of the product delivered by the contractor and net on 
the steps taken or procedures used to provide that 
quality.... It includes appropriate use of preplanned 
inspections, validation of customer complaints, and 
unscheduled inspections. It provides a structured 
appproach to surveillance that permits management 
ccnticl of quality assurance. 



The contractor administers quality control over per 
nier.ee to ensure that a minimum level of service qualit 
maintained. Quality is top management's requirements 
service outputs which stress that these be provided 
cost less than their value. Performance processes mus 
conducted with efficiency and effectiveness. Control re 
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tc procedures of top management used to determine that 
service activities are being carried out in a manner that 
was established by prior planning and goal setting. 

Quality control is aimed at the prevention of unsatis- 
factory performance cf CA contract services. Contractor 
programs fccus on developing employee self motivation to 
render acceptable performance of service. Quality control 
leads to increased profits for the contractor and high 
levels cf customer satisfaction, and facilitates increased 
employee productivity in wort performance. [3ef. 29] 

Circular A-76 emphasizes effective contractor quality 
control and government quality assurance by embracing a new 
type of surveillance procedure known as random sampling 
inspection. Traditional planned inspection and 100 percent 
inspection systems are being replaced by these having a 
statistical basis. Statistical sampling techniques increase 
the objectivity of government quality assurance since each 
item cf service has ar. equal chance cf being inspected. The 
number cf government inspectors required for the surveil- 
lance process is reduced, resulting in less inspection 
costs . 

E. THE EVOLUTION OF STATISTICAL QUALITY CONTBOL 

A brief examination of the definition and history of 
statistical quality centre! will be described. A definition 
cf statistical quality control (SQC) follows: [Ref. 30] 

Statistical quality control is the employment of statis- 
tical principles and methods which have been developed 
to assess the magnitude of 'chance cause variation' and 
to detect 'assianatle cause variation.' Variation due tc 
'chance causes' is inevitable while variation due tc 
'assianable causes' can usually be detected and 
corrected by aporopriate methods. Statistical quality 
control philosophy is the early detection of assignable 
causes so that product quality may be controlled at the 
desired level with a minimum of rejects. 
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to differences in 



Assignable causes of variation are due 
machines, workers, materials, and either their composition 
or relationship to each other over time [Bef. 31]. 

SQC processes contribute to economical achievements of 
product guality by applying a body of theory dealing with 
laws cf large numbers and probabilities to various indus- 
trial and service processes. These originated in 1654 when 
Pascal, a French philospher and mathematician, teamed up 
with Pierre Fermat to develop the theory of probability. 
[Bef. 30] 

Until the 1920's, most semblances of SQC were associated 
with measures of central tendency, or averages. Increasing 
study at this time was devoted to the effect, cf standard 
deviations cn control processes, and led to the foundations 
cf modern statistical quality control. [Ref. 31] 

Walter Shewhart cf Bell Telephone Laboratories developed 
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occurred in World War II due to the need to minimize produc- 
tion scrap losses. Government agencies developed training 
courses for these personnel in industry who inspected 
product or service outputs. After World War II, American 
industry further developed SQC techniques, wnich were 
adopted in European and Asian countries as well. SQC became 
an underlying basis for many industrial productivity 
improvements and is still utilized extensively, as evidenced 
in modern Japanese industries. [Bef. 32] 

Previous discussion may lead one to believe that SQC 
techniques were designed primarily for manufacturing appli- 
cations, but these are equally valid for ncn-manufac-uring 
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types of techniques. The following instances illustrate 
representative applications: [Ref. 31] 

a) 3QC was used by the Census Bureau to control clerical 
accuracy ; 

b) Aiders Inc., a mail order business, utilized control 
charts to establish accuracy in customer billings; 

c) The Illinois Eell Telephone Company used SQC techni- 
ques to assess clerical accuracy in the accounting 
department ; 

d) The Standard Register Company used sampling plans to 
control accuracy of sales invoices; 

e) United Air Lines used control charts to improve accu- 
racy in customer bookings; 

f) Statistical control techniques have been used success- 
fully in the health industry [Ref. 33] ; and 

g) Statistical quality ccntrol techniques have been used 
in highway and airport pavement construction 

[Ref. 34]. 

Eased on the preceding observations, it is reasonable to 
apply statistical sampling techniques to CA service contract 
surveillance programs. Such techniques have been utilized 

by the Air Force in its A- 76 contract conversions. 
Procedures for applying these techniques are stated in Air 
Force Regulation AFR 400-28. 

This regulation incorporates a statistical sampling 
model known as acceptance sampling by attributes. This model 
is more fully described in Department of Defense Military 
Standard 105D of 28 April 1963 ( MIL-S TD- 10 5D) . The standard 
has been used successfully by defense industries since its 
original formulation, and its concepts form the basis for 
worldwide acceptance sampling standards. MIL-STD- 105D can 
be used to inspect the following: [Ref. 35] 

a) End items; 

b) Components and raw materials; 





C) 


Opera 


t ions 


* 
























d) 


Mater 


ials 


in 


P 


recess ; 




















e) 


Suppl 


ies i 


n 


St 


crage; 




















f) 


Maint 


e na n c 


6 


op 


erat ion 


s; 


















g) 


Data 


cr re 


CO 


rd 


s; and 




















h) 


Admin 


istra 


ti 


ve 


proced 


ure s. 
















Th 


X 3 


list 


is nc 


t, 


ex 


haustiv 


e. T 


he 


text o 


f 


MIL -STD- 10 


5D 


is 


F r 


6 S 6 


nted 


in A 


FP 


en 


dix 3. 


?r 


ere 


quisle 


e 


condi’ 


:icns 




for 


3UCC 6 


ss f ul 


use 


cr 




SQC tec 


hni que 


s 


are t. 


hat 


: operation 


s 


be 


-u “ 


pet 


itive. 


in de 


ps 


n d 


ant, fu 


action 


ai. 


and 


ail 


: acted 


by a 


C 




cu 


*si 


de f ac 


rers 


as 


p 


cssibie 


[ Ref. 


31 


]. CA 


se 


srvice 


cent 


r a 


CZS 



meet these require me nts. 

A review of representative SQC literature revealed the 
following objectives: [Ref. 31] 

a) It indicates the presence of assignable causes of 
variation ; 

b) It indicates the specific sources of these causes; 

c) It is as simple as possible; 

d) It leads to remote chances of searching for assignable 
causes of variation when these are not present; 

e) It lowers costs, reducing laoor and materials waste; 

f) It improves quality, making such improvements 

uniformly throughout the entire production or service 
process; 

g) It sets and adjusts tolerances and specification based 
on acquired production experience; and 

h) It improves employee morale and the tenor of 
custc mer- vendor relationships. 

Incorporation of SQC techniques in CA service contract 
administration may initially appear confusing. Proper 

training which presents it in a clear, concise manner should 
preclude this apprehension and lead to acceptance of SQC 
techniques by both government and contractor quality assu- 
rance personnel. ether possible objections are that SQC 
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techniques may not te appropriate for certain business or 
product lines, and managers may mistakenly belive that their 
services would always be performed in an excellent fashion, 
exclusive of the use cf SQC methodology. 

Such objections might be minimized if the advantages of 
SQC are presented. It reduces scrap and rework, increases 
quality awareness in all employees, and enhances produc- 
tivity. Ibis leads to increased quantities of improved 
products and services. Inspection becomes more scientific 
and reliable so that prediction of impending trouble can 
occur. Inspection costs are reduced, while authentic and 
accurate records of guality can be created. 

C. LEGALITY OF STATISTICAL QUALITY ASSURANCE IN GOVERNS ENT 

PROCUREMENT 

1 • Introduction 

A primary goal of government procurement is to 
obtain timely and acceptable delivery of specified services. 
Contractors are alerted to this goal when the government 
includes mandatory clauses which state its rights in 
conducting quality assurance. A typical clause from Standard 
Form 23- A (Construct icn Contracts) is as follows: [Ref. 36] 

All work (which includes but is not restricted to mater- 
ials, workmanshiD, and manufacture and fabrication cf 
components) shall te subject to insoection and test by 
the government at all reasonable times and ail places 
pricr to acceptance. Any such inspection and test is fcr 
the sole benefit of the Government and shall not relieve 
the contractor of the responsibility cf providing 
quality ccntrcl measures to assure that the work 
strictly ccmtlies with the contract requirements. Nc 
inspection or test by the government shall be construed 
as constituting or implying compliance. 

Additionally, other special inspection clauses may amplify 
unique quality requirements cf a contract. 
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The government can exercise great latitude in ins 
inspection cf a contractor's performance, conducting it ir. a 
reasonable manner but increasing its intensity if signifi- 
cant defects in perfcrmance are detected. Unspecified means 
cf testing are allowable as long as these are reasonable in 
conducting surveillance. Methods that increase contractual 
performance should net be utilized; concurrently, government 
surveillance should net interfere with contractor perfcr- 
mance. Inspection by means of sampling prccadures has beer- 
upheld in several cases presented to the Armed Services 
Eoard cf Contract Appeals ( AS BCA) . [ Bef . 36] 

In any inspection system, the government must avoid 
the risk cf effecting constructive changes to the contract 
by unreasonably elevating its own surveillance r eguir aments 
or quality control requirements for the contractor. 

2. S a it r li ng Techniques Substa n tiate d by Boards of 
Con tra ct A ppeals 

SCC techniques have been sustained by the federal 
courts and by federal agency Beards of Contracts Appeals. 
In Vi-Mil Inc., ASECA 16820, 75-2 BCA, para. 11435, 

MIL-STD-105D was utilized to properly estimate quantities of 
defective coats that occurred in several production lots. 
Government inspectors properly organized random samples to 
ascertain contractor perfcrmance and were correct in 
concluding that sample results were representative cf entire 
production lets. 

In Gcldring Packing Company, ASBCA 7736, 1S62 BCA, 

para. 3392, a government decision to terminate for default 
on the basis of sampling results was sustained by the Beard. 
An inspector checked 11 meat loins out 114 total lcins and 
found defects in each sample. 
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The government's choice of acceptable defect levels 
was sustained in Precision Products, ASBCA 14284, 70-2 BCA , 

para. 8447 in regards to sampling inspection. It was made 
clear that such choices must be based on reasonable govern- 
ment reguir ement s; if defects are critical in nature, it is 
allowable to state that no defects or defective performance 
will be permitted. 

with no method of inspection specified in the 
contract, the government proceeds! to use sampling techni- 
ques in assessing product characteristics in Frank and 
Warren, ASBCA 10259, 65-2 BCA, para. 5102. The Board found 
that sample sizes utilized were sufficient to allow a termi- 
nation fcr default. 

In Associate Aircraft Tool and Manufacturing Inc., 
ASBCA 7255, 1963 3CA , the Board stated that: 

where the government purports to reject an entire let of 
items cr the basis or an inspection less than the full 
quantity delivered (the inspection samplina) , the 
inspection sample must either be representative of the 
entire lot, or in accordance with a sampling and control 
plan acreea to in the contract. 

The beard also emphasized that inspectors properly designate 
the manner cf forming inspection lots, the deter miration of 
sample sizes, and the manner in which contractors present 
lots fcr inspection. 

Sampling procedures for inspection of manufactured 
products were included in contract provisions for Metal Tech 
Inc., ASBCA 14828, 72-2, para. 9545. The Eoard sustained 

the manner in which delivered items were rejected and 
sustained the government's termination of the contract, 
citing the Frank and Warren decision which stated that the 
government is not obligated to inspect eaca defect in all 
delivered supplies when forming a basis for lot rejection. 
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Ir. a final case, American Quality Assurance 
Engineering, ASBCA Nc. 114 17, 1 1466, 1 1544, ar.d 1 1747, 63-1 

ECA, para. 6986, the government used a liquidated damages 
clause which stated: 



The government will inspect approximately 10 percent cf 
the tasks performed daily by the contractor. Of these an 
acceptable quality level of a daily average cf net mere 
than eighteen tasks has been estaolished. If fer any 
calendar month the contractor exceeds this average he 
shall pay to the government as liquidated damages for 
excess administrative costs the sum of 3250.00. The 
daily average of unsatisfactory performed tasks is 
obtained by dividing the total of unsatisfactorily 
performed tasks for the month by the number of working 
days for the month. 



In this case, government payment withholdings ware in 
dispute. The government based these on judgmental samplings 
(not random) . Some withholdings were sustained while ethers 
were denied by the Ecard. Deductions from total monthly 
invoices based on defect percentages observed in the sample 
were found to be correct. The Board disagreed with enlarging 
observed percentages to 100 percent of the projected defi- 
ciencies, and applying these to the entire month's perfor- 
mance; it believed that this enlargement was conjectural and 
lacked sufficient accuracy. Utilization of Mil Std 105D 
sampling techniques might have injected sufficient accuracy 
into the surveillance process, leaving nc room for dcubts 
about the propriety cf deductions. 

3 . GAO Sub sta nt i ation of St at istical Qualit y Assu ranc e 

Techni gu es 

A recent General Accounting Office (GAO) ruling 
substantiates government inspection by statistical sampling 
methods. In Environmental Aseptic Services Administration 
and Larsen Building Care, B-207771 of 28 February 1983, GAO 
evaluated protests against Air Force implementation of 
extrapolated deductions using MIL-STD-105D. An issue of 



dispute was the definition of work items, as each one might 
consist of several subsidiary tasks. For example, cleaning 
cf rooms could involve sweeping floors, emptying trash cans, 
dusting blinds, and several other tasks. Because of the 
deduction provisions cf an Air Force CA contract solicita- 
tion, the protestors claimed that one defective subsidiary 
task could lead to rejection cf the entire room just as if 
all subsidiary tasks were judged to be unsatisfactory. 

The GAO agreed, finding that the government require- 
ments were not fair cr reasonable and could be viewed as a 
penalty system. GAO recommended that the Air Force distin- 
guish between vital and non-vitai tasks, establishing 
reasonable deduction rates for both. This has resulted in 
more detailed breakouts of work requirements for certain Air 
Force CA contract solicitations. 

The GAO emphasized that enclosing the quality assu- 
rance provisions in solicitations was clearly for the 
benefit cf the government and not potential offerors. These 
can not be disputed, nor can failure by the government to 
adhere tc them fcrm a basis for protest. 

4 . Sum mar y 

The preceding discussion points our a sample of 
legal cases in which government use cf statistical qualify 
assurance techniques were disputed. These were found to be 
valid and applicable to government procurement. These tech- 
niques must be reasonable and net increase stated standards 
cf performance. Use of 3QC techniques are the prerogative of 
the government and net the contractor. Rejections or deduc- 
tions based on sampling must be representative of the lets 
cf werk cr service being performed. 

SCC techniques have a legal basis when applied to CA 
service contracting. Ensuing discussion will examine 
various statistical methods which are being used or are 
being studied for their possible application. 
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D. AIR FORCE CA CONTRACTING UNDER AFR 400-28 AND 

HIL-STD-105D 

1 . Intr odu ction 

The Air Force first issued policy recurring the 
concurrent design of performance-oriented statements of work 
and matching quality assurance plans for CA service 
contracts in Air Force Regulation AFR 400-28. Subsequently, 
standardized statements of work and quality assurance plans 
which could be tailored to local command needs were devel- 
oped and distributed. AFR 400-28 required the use of random 
sampling procedures based on MIL-STD- 105D, along with ether 
inspection methods. Other executive agencies followed the 
Air Fcrce lead. The Office cf Federal Procurement Policy 
issued OF PP Pamphlet No. 4 , which embodies the procedures 
stated in AFR 400-28. The Naval Facilities Engineering 
Command issued its cwn maintenance manual, MO-327, which 
calls for performance-oriented statements of work but dees 
net implement the use cf MIL-STD-105 D. NAVFAC is exploring 
the use of another statistical method, confidence level 
estimation, which is examined later in this study. 

2. M i 1 1 ta r y Standard 1 05D C once p ts an d issues 

Eefore examining specific Air Force SQC policy, it 
is necessary to elaberate on the basic statistical concepts 
and issues cf MI L-STE-105D. These are based or. acceptance 
sampling and are described in simple terms by A. J. Duncan, 
a noted SQC author: (Ref. 32] 

A company receives a shipment of goods. It samples the 
shipment and either accepts it as conforming to stan- 
dards cr rejects it. If the company rejects the lot as 
beina fcelcw standard, it may be returned to the supplier 
or if may be kept, depending cn how badly the aoocs are 
needed cr what arrangements have been made with the 
supplier. Possibly there will be a price concession on 
rejected lets. It is to be emphasized that the purpose 
of acceptance samcling is to determine a course .of 
action, hot to estimate lot quality. Acceptance sampling 
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specifies a procedure chat, if applied to a series of 
lots, will give a specified risk of accepting lets of 
given quality. 



In ether words, acceptance sampling 
ranee. It is alsc emphasized cnat a 
not an attempt to 'control* qualit 
purpese cf control charts; these 
modifying procedures so as to turn 



yields quality assu 
cceDtance sampling i 
y. ‘The latter is th 
guide the engineer i 
out better products. 



n 



Under acceptance sampling, the attributes cf a 
product are judged. Attributes are that property of a unit 
that classify it as tad or good. Quality characteristics of 
a unit are either within specified limits, or are not 

cen forming. [Ref. 11 ] Submission of good quality services 

results in high rates of acceptance, while products of poor 
quality incur a high rate of rejection. 

For CA contracts, single sampling plans fer frac- 
tions defective are used most frequently. These define a 
sample size that is to be taken and a number of defective 

units which can not be exceeded in order to prevent lot 

rejection. [Ref. 32] As an example, a sampling plan may 
call for a sample of 100 CA service work items to be taken 
from a monthly lot cf performance. Two or less defectives 
resuit in let acceptance, while three or mors lead to lot 
rejection. Such stated constraints lead to the construction 
of an operating characteristic curve, which illustrates how 
the probability of acceptance of a lot varies with the 
quality cf the material offered for inspection. At lew rates 
of discovered defects, the probability of lot acceptance 
will be high. At high rates of discovered defects, the prob- 
ability cf let acceptance will be low. Operating character- 
istic curve profiles can be adjusted by varying lot and 
sample sizes or by varying the acceptable defect rates 
(acceptable quality level, AQL) as shown in Appendix C. Such 
operating characteristic curves illustrate the protection 
offered to both ccntractcrs and government customers. 
Application of military Standard 105D requires the following 
sequence cf planning activities. 
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Lot Sizes for MIL-STD-1 05 D QA Plans 



In using random sampling procedures and 
MIL-SID- 1 0SD guidelines, the size of work item lets must 
first be determined. Lots could be the total number of roems 
in a building that are cleaned, the number of vehicles that 
undergo preventitive maintenance or the number of refuse 
containers that are serviced on a monthly basis. Lots can be 
accumulated on ether than a monthly basis, but should be 
repetitive cr continuous in nature. All let items should be 
homogenous, cr have the same characteristics. 

fc. Sample Size Determinations 



When a let size 
are consulted to determine 
judging the characteristics 
there are three different 
three levels of inspection 
when smaller, or reduced. 



is lenewn, tables in KIL-STD-1G5D 
an appropriate sample size in 
of the lot. For one let size, 
sample sizes corresponding to 
intensity. Level I is utilized 



sample sizes are sufficient; less 



discrimination is necessary. 

Level II is the normal level of inspection 
intensity. Sample sizes derived from Level II tables are 
used most frequently for CA contracts. 

Level Ill's larger inspection samples are used 
when more discrimination of product quality is necessary, 
resulting ir tightened inspection. These are used when there 
are major observed declines in product quality. 

c. Determination of Acceptable Quality Levels 

The next activity in the sequence of designing a 
quality assurance plan is the determination of an Acceptable 
Quality Level (AQL). This variable is described 
MIL-SID-105C as follows: [ Bef . 35] 



13 



The AQL is the maximum percent defective (or the maximum 
number cf defects per hundred units) that, for -.he 
purpose of sampling inspection, can be considered satis- 
factory as a process average, when a consumer designates 
some specific value of AQL for a certain defect or arcuc 
of defects, he indicates to the supplier that his' (the 
customer's) acceptance sampling Plan will accept the 
great majority of lots or batches that the supplier 
submits, provided the process average level of percent 
defective ‘(or defects per hundred units) in these lets 
or batches is no greater than the designated value of 
AQL. Thus, the AQL is a designated value of percent 
defective (or defects per hundred units) that the 
customer indicates will be accepted most of the time by 
the acceptance sampling procedure to be used. The 
J ’ms provided n ere in are sc arranged that the 
, of acceptance at the AQL value depends upon 
tns sample size, being generally higher for larae 
samples than for small ones for a given AQL. The AQL 
alone dees not describe the protection for individual 
lots or batches but more directly relates to what might 
be expected from a series cf lots or batches, provided 
the sieps indicated in this publication are taken. It 
is necessary to refer tc the operating characteristic 
curve of the plan to determine what protection the 
consumer will have. 



sampling plans 
peccability 



The designation of an ACL shall net imply that the 
supplier ras the right to knowingly supply any defective 
unit of product. 

The AQL tc be used will be designated in the contract or 
by the responsible authority. 

The values of AQL's given in these tables are known as 
preferred AQL's. If, for any product, an AQL be disig- 
nated ether than a preferred AQL, these fables are not 
applicable. 



Duncan states the following in reference tc the 
AQL: [Ref. 32] 

In apclyina the J1IL-STD- 105D it is expected that in a 
conference between a supplier and a military agency it 
will be made clear to the supplier what, for purposes cf 
acceptance sampling, the agency considers to be accep- 
table quality levels for product characteristics. 

With a specified AQL and inspection level inten- 
sity, the sample size can be determined by using tables 
given in J1IL-STD-105E. These also allow a determination of 
accept and reject numbers from tnese tables. The a ccep t 
numb er is an important threshold; as long as the number of 
defectives found in a sample are less than or equal tc this 
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value, the let is accepted. If 
equal tc or greater than the 
rejected . 

d. Inspection Schedule 
Tables 



the number of defectives are 
r ej ect n umber . the lot is 

Design Usrng Random Number 



The next step in the quality assurance plan 
design is the choice of items that will be included in a 
sample by using random number tables. The process begins by 
assigning each item cf work in a certain functional category 
(such as service of housing area trash containers) its own 
unique inventory number. 

A random number table is then consulted. 
Ccrr espcndence between work elements in the lot and digits 
listed in the table is established (a numbering system). A 
routs through the table is selected and is followed by 
choosing numbers according to this pattern. A starting point 
is fixed, and the table is used until the required number of 
sample items is chosen. 

In using random number taoles, the selection of 
inspection sample items occurs such that each has an equal 
chance cf being included in a sample. Detailed, explicit 
instructions on random number table usage are provided in 
AFR 400-28, OFPF No. 4, and NAVFAC MO-327. The quality 
assurance inspector lists each rtem on a schedule and 
conducts inspections at the appointed time. Results are 
documented in writing to aid in determining the accept- 
ability or rcn-acceptabilit y cf contractor performance. 



e. The Disposition of Inspection Results 

After inspection findings are complete, alterna- 
tive courses of action may be pursued by the activity 
contract inspection division. When AQL's are exceeded, 
payment deductions or lot rejections may be made. If two of 
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five consecutive lots are rejected (AQL's are exceeded) , 
tightened inspection is instituted utilizing larger sample 
sizes. Rejected lots may be resubmitted after defects are 
corrected, but this is done only at the discretion of the 
government. 

When ten consecutive lots subject tc tightened 
inspection are rejected, performance is halted. It is 
conceivable that a termination could ensue cn or before this 
point. There is a provision for exceptional contractor 
performance. When ten consecutive lots subject tc normal 
inspection have been accepted, a switch to reduced inspec- 
tion (smaller sample sizes) may be made. 

f. Overview 

If quality assurance plan designs based on the 
preceding sequence cf activities are conducted properly, 
MIL- STD- 1 C5D will provide for effective and reliable 
surveillance of CA service contracts. 

The concept of acceptable quality levels must be 
carefully understood. It is not an aspect of specifications 
or cf performance. It is instead a notification to contrac- 
tors that government surveillance plans will allow no amount 
cf -defects observed in the sample to be greater than the AQL 
specified by the government. [Ref. 32] 

The most commonly used table in the standard is 
the single sampling plan (Table 1 1 - A ) . It is designed so 
that alcnc diagonal path in the table, the product of the 
AQL ana the sample size is nearly constant. T has has 
resulted in a limited set of AQL's which may be utilized, as 
snown in Appendix 3. 

Cne criticism of MIL-STD-105D is that probabili- 
ties cf acceptance increase as sample sizes increase for one 
given level of AQL. When the standard was being formulated, 
some industrial engineering experts pressed for constant 
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AQL's are standardized values which are jointly 
determined by the field activity or its systems command 
headquarters. As an example, :icClelland Air Force Base 
divisions submit prospective CA solicitations with desired 
AQL's tc Air Force Logistics Command (AFLC) headquarters in 
Dayton, Chic. AFLC headquarters either approve the activity 
choices of AQL's, or recommend changes. AQL's given ir. 
MI L-S IE- 105C tables must be utilized. AFF 400-28 calls for 
AQL selections that are realistic in helping tc secure a 
minimum quality of service, since no service is capable of 
being perfectly performed. These are then communicated to CA 
contract bidders in a form entitled the Performance 
Eeguirements Summary (PES) as shown in Appendix C. It lists 
each element of required performance, the standard for its 
performance, the maximum allowable deviation from this 

requirement, and the method of surveillance that will 

adjudge performance quality. [Ref. 39] 

The contractor determines the percentage of each 
individual category of work in relation to the total 
contract value, entering these on Performance Requirements 
Summary sheets and returning them with the bid submission 
package. Such percentages are later used in making deduc- 
tions for unacceptable performance. 

It should be noted that the use of MIL-STD- 105D may 
allow observed defects rates greater than the specified AQL 
whan the contractor reaches cr exceeds the reject number. 
For example, a lot of 2000 items is checked with normal 
sampling intensity (level II) and an AQL of 10 percent. A 
sample of 125 items is required; the lot is to be rejected 
if 22 defects are observed in the sample. If 22 defects are 
later discovered, the observed defect rate (17.6$) exceeds 
the AQL (10$) and rejection would occur. Sven though any 
defects greater than 13 would cause the observed defect rate 
to exceed the AQL cf 10$, this method allows up tc 21 
defects tc be accepted. 
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The Air Fores has adopted a policy of assisting 
contractors in developing their own quality control 
programs. Accordingly, activities who issue CA solicitations 
enclose a copy of the quality assurance plan. This enclosure 
is marked as follows: [Sef. 39] 

For Information Purposes Only. This Quality Assurance 
surveillance Plan ns net part of the Request for 
Proposal cr Invitation For Bias nor will it be made cart 
of any resulting contract. 



A Contract Administrator plan can also be enclosed 
with the soli citation, describing Contract Administrator 
duties in evaluating the performance cf Quality Assurance 
Evaluators (who inspect the contractor). Contract 

Administrators may also make random inspections of contract 
per f crmance . 

Contractors are provided appropriate pages from 
aiL-STD-105C to assist them in establishing their own 
quality control systems. Instructions are provided which 
describe hew extrapolated deductions will be mads when 
specified AQL' s are exceeded. The contractor is never given- 
schedules cf inspection which have oeen developed from the 
randem number tables. 

The Air Force approach appears to be one of reason- 
ableness. Conversations with personnel at two Air Force 
bases revealed that performance rendered under CA contracts 
with statistical surveillance methods is very satisfactory. 
Most problems occur in the initial transition periods of 
contracts (the first one or two months of performance). 
AQL 1 s fer most contracts at both installations have rarely 
been exceeded. Government personnel seemed to be pleased 
with the results of random sampling inspection methods. 
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Sens deviations from MIL-STD-105D are detected in 
AFR 4C0-28. No provision is made for tightened inspections, 
as single sampling tables in the regulation address only 
normal (Level II : average quality) and reduced (Level I : 

good guality) inspection. 

Switching from normal to reduced inspection under 
AFR 400-28 is allowed when four consecutive lots have been 
accepted, and the number of defects is less than one half of 
the specified acceptance number for normal inspection. The 
division manager and contract administrator must also agree 
tc the switch. MI L-SID- 1 05D allows this only aftsr ten 

consecutive lots have been accepted. A return tc r.crmal 
inspection will be implemented if the acceptance number for 
reduced inspection is exceeded. Under MIL-STD- 105D , a 
switch tc tightened inspection is necessary when two of five 
consecutive lots are rejected. A switch from tightened back 
to normal inspection is allowed when five consecutive lots 
are accepted. AFR 400-28 does not incorporate a similar 
provision . 

It might be reasoned that these Air Force deviations 
are meant tc effer positive motivation to contractors in 
performing satisfactorily. These modified procedures are 
being utilized at most Air Fcrce activities undergoing A-76 
CA contract evolutions. 

E. NAVFAC USE CF CONFIDENCE LEVEL ESTIMATION METHODS 

The Naval Facilities Engineering Command (NAVFAC) has 
been heavily involved in the management of a growing number 
of CA contracts for facility related services at Navy activ- 
ities. It is one of the first systems commands under the 
Chief of Naval Material (CNM) to address diverse issues in 
CA conversions and begin the formulation of policy to deal 
with them. 
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NAVFAC published the MO-327 to offer guidance to Navy 
activities in preparing performance-oriented statements of 
work and quality assurance plans. This guidance is similar 
to that provided in Office cf Federal Procurement Policy 
Pamphlet No. 4, and allows inspection methods ether than 

random sampling. NAVFAC has issued 20 standardized state- 

ments of work with matching quality assurance plans which 
can be tailored by Navy activities to incorporate specific 
local requirements. Portions of MO-327 are illustrated in 
Appendix D. 

NAVFAC has not adopted the use of MIL-STD-105D as the 
basis for its statistical sampling techniques, and does not 
yet allow Navy activities to maxe extrapolated deductions 
based on its usage. Sampling techniques may be used at 
those Navy activities which demonstrate the ability to 
establish sophisticated quality assurance programs. New PWC 
transportation CA solicitations being issued in this fiscal 
year will include modified sampling techniques and extrapo- 
lated deductions provisions based on their usage. 

NAVFAC philosophy is that CA contract surveillance 
should be based on the statistical estimation of defective 
items in samples, rather than on the accept/re jset 
hypothesis testing methodology of MIL-STD- 105D. Estimation 
is intended to inject a higher level of precision (confi- 
dence) in the surveillance process than that offered by 
MIL-STD- 105C. 

This methodology requires a designation of desired 
confidence levels, a relative accuracy of estimation, popu- 
lation sizes, and a threshold of conformance (equivalent to 
the AQL concept). Combining these elements results in a 
determination of sample sizes and corresponding lower confi- 
dence limit rejection numbers. For example, a work function 
is performed 2000 times in a month. A confidence level cf 95 
percent is desired in estimating the number of population 



59 



defects, with a relative accuracy equal to 50 percent of the 
designated AQL cf 1 0 percent. (Relative accuracy then equals 
5 percent). A sample size of 93 items is required, and 
rejection of the month’s performance can occur if more than 
16 defective items are found. Sixteen observed defects will 
result in proper let rejection since the estimated lower 
confidence limit percentage for nonconforming items is 
always greater than the specified AQL of 10 percent. 

NAVFAC has worked with the Office of Naval Research 
(ONR) in developing a set of new confidence level estimation 
tables which can be used to design CA contract quality assu- 
rance systems. These tables will reflect three different 
levels of inspection intensity (corresponding to various 
confidence levels) , known as tightened (99 percent confi- 
dence) , normal (95 percent) , and reduced (90 percent) 
sampling. It appears that these new tables are based on the 
hyper geemet lie statistical distribution. 

This new methodology emanated since NAVFAC doubts that 
MIL-S1D-105D suitably estimates fractions of nonconforming 
activities with reasonable accuracy. This was illustrated 
earlier when observed defect rates at the MIL-STD-1C5D 
reject number were compared to the AQL's. NAVFAC also 
desires flexibility in choosing AQL's other than these 
provided in the standard's tables. 

A primary emphasis in confidence level estimation is to 
determine the actual occurrence of defects in the population 
based on sample observations whereas hypothesis testing by 
attributes only determines if populations are acceptable or 
unsatis f act ory . 

NAVFAC has ether reservations concerning the use of 
MIL-STD- 105 C. It does not believe that full payments should 
be made when the observed defect rata exceeds the AQL 
percentage, even if the reject number has not been exceeded, 
and questions the use of the standard where lots may net be 
homogenous and be submitted in a continous manner. 
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ether differences in Air Force and NAVFAC contract, 
policy were observed during the study. The Air Force 
encloses copies of quality assurance plans with CA contract 
solicita tic ns; NAVFAC contracting activities have not dene 
so. Air Force contractors are 'notified of desired AQL's and 
the type cf inspection method used in verifying these. 
NAVFAC will provide this information tc prospective bidders 
in its future CA solicitations. The Air Force administers a 
management control program in which cognizant Contract 
Administrators conduct random checks of both inspector and 



TABLE I 

Samples Sizes For UIL-STD-105D and Confidence Intervals 



Lot Size 



50 
100 
5 CO 
1000 
5000 
10000 



S ampl e sizes j 

a IL- STD- 1 0 51) Confidence Intervals ! 



8 

20 

50 

80 

125 

125 



37 

58 

108 

121 

135 

136 



These samples sizes are for normal inspection at 
95?! confidence, at an AQL of 1QX. 



contractor performance. NAVFAC has considered such a 
program but has not formerly instituted one. 

The most noticeable difference in inspection philoso- 
phies is visible if sample sizes of each are compared over 
increasing let sizes, as shewn* in Table I. Sample sizes 
under MIL-STD-105D are less than those for confidence level 
estimation for smaller populations. Two inferences can be 
drawn from this fact. If cost savings are cf paramount 



61 



importance in conducting CA contract surveillance, then 
HIL-STD-105D sampling tables should be utilized because 
smaller numbers of required inspections result in less costs 
to the activity. However, if precision in adjudging 
contract performance is essential, then sampling under 
confidence level estimation should be utilized. The larger 
sample size provides greater reliability in assessing the 
number cf defects that might be submitted. If costs of 
inspection are net great, this method is preferable. 

In one study at the Charleston Naval Shipyard, random 
sampling procedures were used successfully for daily, repet- 
itive services. All inspection personnel found these to be 
superior tc former heavy reliance on planned sampling 
methods. Additionally, use of such techniques resulted in 
significant lowering cf observed defect rates in contractor 
performance . 

The authors were told that an informal DOD working panel 
of Air Force, Navy, and OFP P personnel will begin an evalua- 
tion cf different available statistical methods when the new 
CNR sampling tables are complete. Until then, the only offi- 
cial statistical method available for use is MIL-STD- 1 0 5D . 

F. USING STATISTICAL SAMPLING RESULTS FOR EXTRAPOLATED 

FAYSENT DEDUCTIONS 

As well as promoting more optimum use of costly inspec- 
tion resources, SQC techniques provide another important 
benefit. Their results can be utilized in extrapolating, or 
applying the percentage of defectives found in samples to 
the larger let populations. 

The Air Force first mandated this procedure in AFR 
400-28. fchen sample errors occur at a rate which is greater 
than the desired AQL and its corresponding reject number, a 
percentage -found unacceptable results. It is subtracted from 
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100 percent to determine the lot percentage acceptable for 
payment, which is applied to the entire lot. Deductions are 
not taken when errors in a sample are less than the reject 
number. 

NAVFAC policy has not allowed extrapolated deductions 
since it has no official statistical method on which to base 
these deductions; however, progress in developing such 
methods has led NAVFAC to envision alternative applications 
of extrapolated deductions to be utilized when a uniform DOD 
statistical sampling colicy is adopted. Appendix E illus- 
trates various Air Force and NAVFAC methods of making 
payment deductions fcr noncorf crming service. The NA7FAC 
methods have not yet been promulgated as new CA contract 
policy. It should be noted that NAVFAC will deduct for all 
items observed to be in nonconformance. When contractors 
sati sf actor ily reperfcrm these within allowable time frames, 
however, credit will be given for reperformed items. 
Liquidated damages are assessed as a percentage of all 
ncnccnfci ming items cf work. 

Promulgation of a uniform DOD statistical sampling 
policy will play an important role in allowing the use of 
extrapolated deductions based upon this technique. If 
sampling procedures are perceived to be unreliable, litiga- 
tion proceedings may eventually prohibit the use cf this 
deduction methodology. 

G. AN OVER VIEW OF SQC TECH NIQOES FOR A-76 CONTRACTS 

It should be apparent that the use cf random sampling 
procedures in the new A-76 Commercial Activities program for 
COD and other federal agencies offers significant cost 
savings in contract administration. Inspection costs are 
reduced, ard incorporation cf SQC techniques may lower the 
costs that contractors incur in performing such services. 
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These techniques cffer fairness and objectivity Thai 
cften were net present under former planned sampling and 
customer complaint programs. If SQC techniques are adminis- 
tered properly, they will render a much more accurate 
presentation of true contractor performance than was 
possible under former inspection methods. 

Such techniques will be invaluable if large, multifunc- 
tion contracts at federal activities become a reality (such 
as the total base operating services contract at the Naval 
Submarine Base Bangor, Washington). These will optimize the 
use of minimal government contract administration resources. 
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v* JIANNI NG AND EODGET ING FOR CO NTR ACT INSPEC TIO N 

ORG ANIZATIONS 



A. INTRODUCTION 

With the implementation of OMB Circular A-76, DOC activ- 
ities that study CA functions must prepare for the possi- 
bility of a contract award by identifying and planning for 
certain support requirements that would be nece ssitated. 
Such plans include a quantification of inspection resources 
so that the activity's budget can accurately reflect fund 
requirements for either the creation of a service contract 
organization or for the augmentation of an existing one. 

The Commander, Naval Facilities Engineering Command 
(NAVFAC) and the Commander, Naval Supply Systems Command 
(NA V S UP) have traditionally held contract authority for 
construction and procurement of supplies. Both have dele- 
gated contract authority to regional procurement commands 
such as NAVFAC's Engineering Field Divisions (EFD's), 
NAVSUE's Navy Regional Contracting Centers (NRCC's), and 
Naval Supply Centers (NSC's). These commands have further 
delegated contract authority to field activities. Along with 
this authority , field commands have been assigned the 
responsibility for guality assurance and surveillance of 
construction and delivered supplies . In the Naval 
Facilities Engineering Command, these are known as Resident 
Officers In Charge of Construction (ROICC) . These field 
activities have been staffed with necessary quality assu- 
rance and contract specialist personnel to administer 
construction and supply contracts. Additionally, the 
responsibility for the preparation of contract solicitation 
packages (for example, the design of construction projects) 
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has usually been assigned to the regional contracting office 
cr to the requiring activity. 

This pattern of contract authority and surveillance 
responsibility does not apply, however, for CA service 
contracts. Neither NAVFAC nor NAVSUP have been provided wrth 
sufficient staffing resources to provide on-site surveil- 
lance and administration of contracts resulting from A-76 
cost studies, although both systems commands retain contract 
authority fcr CA contracts. Navy activities that receive 
services under these contracts prepare the statements of 
work, quality assurance plans, and provide an on-site 
surveillance organization to ensure proper contractor 
performance. 

A Chief of Naval Operations message that was promulgated 
on 22 November 1982 delineated different roles fcr both 
contracting agent commands and customer activities in their 
implementation of the CA program. The following responsi- 
bilities are identified; [ Bef . 37] 

a) Contracting Office 
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b) Customer Activity 

i) Provision of qualified personnel to inspect work 
delivered under the contract, 

ii) Preparation and implementation of quality assu- 
rance plans, 

iii) Submission of quality assurance summary reports 
to the contracting agent, 

iv) Evaluation of contractor requests for payment, 

v) Recommendation of payments or deductions to the 
contracting agent, 

vi) Submission cf cost estimates to the contracting 
officer for proposed modifications, 

vii) Provision of assistance as required to the 
contracting agent during modification negotia- 
tions, ar.d 

viii) Performance of other service contract support 
duties when these are delegated by the 
contracting office. 

NAVFAC has alerted all Navy field activities and major 
claimants of this division in service contract responsibili- 
ties in both an instruction (NAVFACIN3T 4330.45) and a new 
maintenance manual (NC-327) . The Air Force has adopted a 
similar approach which is specified in two separate Air 
Force Regulations (AFR 400-28 and AFR 70-9). They require 
that the functional divisions of each Air Force base prepare 
both the statement cf work and corresponding quality assu- 
rance plan. The division must provide all necessary quality 
assurance personnel and, prior to the contract solicitation, 
certify in writing to the base Commanding Officer that these 
personnel have been designated, trained, and dedicated 
solely to perform surveillance functions. [Ref. 38] 

At two Air Force installations visited during the 
research phase of this thesis, it was discovered that a 
consolidated steering committee is organized upon the formal 
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announcement of a CA cost study. It includes representa- 
tives from all departments that might be affected by the 
contracting out of a function and tailors standardized Air 
Force statements of work that have been prepared for each CA 
function, identifying and including all special local 
command requirements. The steering group also provides 
assistance in the development of the quality assurance plan. 

B. TEE ESTIMATION OF REQUIBED INSPECTION RESOURCES 

At the cutset of a cost study announcement, an activity 
must begin to define and plan its quality assurance require- 
ments. [Bef. 39] The planning and budgeting responsibility 
for obtaining inspection resources belongs to the activity. 
Each Navy activity should immediately begin the planning for 
a contract administration staff, establish ongoing interface 
with the appropriate major claimant to document these 
requirements, estimate future budget amounts, and obtain 
ceiling pcints to facilitate the creation of this vital 
organization . 

Various procedures have been used for determining the 
required quantity cf service contract administration 
resources. In the supplement of the 1979 revision of A- 76, 
it was stated that ccsts of administration resources were to 
be estimated to be four percent of the projected award ccst 
cf a service contract. This factor is useful only as an 
approximate estimate of inspection requirements. Actual 
inspection costs might range from tan percent for small 
service contracts to only two percent for large BOS 

contracts. Use of this simple estimating factor does not 
account fcr differing degrees of complexity among contracts; 
however, if may be satisfactory as an initial estimating 
tool. [Bef. 5] 
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Another simple estimating model for determining inspec- 
tion requirements has been recommended by the the Southern 
Engineering Field Division (SOUTHDIV) of the Naval 
Facilities Engineering Command. It recommends that activi- 
ties can plan to use 875 hours of inspection for every 
$300,000 dollars of refuse collection service contracts, 
while all ether service contract types raguire at least 
1,325 hours of inspection for every $300,000 of contract 

TABLE II 

New A- 76 Contract Administration Factors 



; Staff Positions 


Contract Administration 


[ Studied 


Staff Requirements 


Below 10 


Use existing staff 


10 - 20 


1 


2 1 - 42 


2 


43 - 65 


3 


66 - 51 


4 


92 - 119 


5 


120 - 150 


6 


15 1 - 184 


7 


135 - 222 


8 


223 - 265 


9 


266 - 312 


1 0 


3 1 3 - 367 


1 1 


368 - 429 


12 


430 - 5 CO 


13 


501 - 563 


14 


584 - 682 


15 


683 - 300 


16 


A b o ve 800 


2 % of In- House Staff Est 



costs. This model, lake the previous one, is primarily 
useful for making rudimentary projections of service 
contract inspection requirements. 

The 1983 proposed revision of Circular A-76 provides 
guidance shewn in Tabla II that relates the number of 
personnel required fer contract administration to the number 
of personnel positions being studied for ccnversicn. It 
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shows hew tc estimate the required contract office staffing. 
This particular methodology, as the two previous ones, may 
not account for differing complexities which exist in 
service contracts. 



C = 69.74 + (0.183) (A) + (7.29) (E) 

Where A = Total Service Contract Value (5,000) 
and B = Total # of Service Contracts 
QAS's required - C divided by 144 



Figure 5.1 Inspector Hours Regression. 

Another method for estimating the number of required 
inspectors is found in the Student Guide for Maintenance 
Service Contracts published by the Civil Engineer Corps 
Officers School. This mathematical method, depicted in 
Figure 5.1, is an estimating model that relates the number 
of required inspectors to the total monetary value and 
number of the activity's service contracts. [Ref. 40] 

The Atlantic Engineering Field Division (LANTDIV) of the 
Naval Facilties Engineering Command has proposed an alterna- 
tive which ray be the most viable one recommended thus far 
and is provided as Appendix F. LANTDIV Instruction 11014.4D 
recommends the use of a standardized worksheet to accurately 
estimate required inspection resources. It requires the 
specific quantities of various work elements to be performed 
under the contract on a monthly basis, estimated number of 
inspection hours to observe these, and the total number of 
hours tc inspect individual categories as well as the total 
contract performance fer each month. This estimation will 
allow a determination cf the required number of inspectors, 
based upon a total estimate of inspection manhours. 
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Utilization of this worksheet should result in mere 
accurate calculations of inspection personnel resources that 
are required if a new contract is awarded. This approach may 
also lead to the creation of sufficient data that can be 
used in the development of engineered performance standards 
for inspection of commercial activities service contracts. 

If any of the above estimating models prove to be unsa- 
tisfactory, then the safest means of estimating personnel 
requirements may be to predict that one new inspector will 
be necessary for each new function to oe awarded. This may 
be especially true if the inspection forces are located 
under a functional department, as is often the case for Air 
Force service contracts. 

Inaccurate estimates of required staffing may create 
serious problems for activities that continue to contract 
cut under A-76 guidelines. Navy activities must carefully 
design all of their quality assurance plans and calculate 
the required number of inspectors and staff personnel to 
execute them. Although current policy dictates that defini- 
tized quality assurance plans and organized quality assu- 
rance staffs will be established before contract 
solicitations, research for this study indicated that formal 
quality assurance plan designs often lag contract award. 
Activities that were visited frequently used existing 
personnel to provide surveillance on the new CA contracts. 

Action pursued in accordance with thsse considerations 
will be essential to safeguard the activity's interest in a 
growing trend of contracting out for the Navy's commercial 
services. If this evolution is not planned properly , inef- 
fective or erratic government surveillance may result. Such 
a condition can be a significant factor in these contract 
litigations or disputes which arise when the propriety of 
government inspection is a major issue . 



C. QUALITY ASSURANCE TRAINING 



Up -o this point, it has been emphasized that the deter- 
mination and staffing of a sufficient number of contract 
inspection and administration personnel is vital to a 
successful i irple mentation of A-76 objectives. An equally 
important consideration is appropriate training cf these 
personnel who will be involved in contract surveillance. 
Even if no additional inspection personnel can be obtained 
through budgetary procedures, the activity will still be 
required to provide a cadre cf inspectors from existing 
personnel assets who will need to be knowledgeable in 
surveillance techniques. [Ref. 41 ] 

During this research, it was discovered that two Air 
Force activities designate and train Quality Assurance 
Evaluator (QAE) candidates during the statement cf work 
preparation process in accordance with Air Force Regulation 
70-9. Current Navy policy has left the choice of inspectors 
and training responsibility to the discretion of each activ- 
ity’s Commanding Officer. Navy guidance in this area (that 
of NAVFAC and NAVSU?) has been only advisory in nature. The 
only written mandatory requirements levied upon Navy activi- 
ties is that they prepare and submit a quality assurance 
plan to contracting agents for approval prior to solicita- 
tion; they must also certify in writing that a quality assu- 
rance workforce will be established or augmented prior to 
the contract award. Ihese requirements have been addressed 
in recent NAVFAC Engineering Field Division guidance that 
pertains to CA service contracting. 

Newly designated Quality Assurance Evaluators cannot be 
expected to immediately and professionally execute their 
surveillance responsibilities until they have been qualified 
by means cf a formal training process. This may require 
attendance at a special school designed to teach various CA 
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quality assurance r esponsib ilities. It may also involve an 
ongoing, less formalized training process that is carried 
out by skilled quality assurance specialists who reside at 
cr visit the activity. 

Service contract training courses have been designed and 
are being taught to the growing number of federal personnel 
who perform surveillance functions. For the Air Force, 
training is conducted by a quality assurance program coordi- 
nator. The QA Program Coordinator serves in a general advi- 
sory capacity and monitors the performance of functional 
department CAE's after they complete initial training and 
begin performing surveillance duties. QAE candidates receive 
basic training in contract law and administration, quality 
assurance duties and re spo nsibilities , and an overview of 
the quality assurance plan . The training responsibility for 
Quality Assurance Evaluators is placed at the activity level 
under Air Force procedures. Quality Assurance Evaluators are 
identified and trained before the contract solicitation 
process begins . 

For the Navy, Quality Assurance Evaluator manning is 
handled by the activity and the training is carried cut by 
the nearest Engineering Field Division. The Facilities 
Division (code 10) of each NAVFAC SFD provides this training 
to naval activities in its geographic area. Training of Navy 
QAE's is a contracting agent responsibility. EFD's also 
provide technical support to each activity during the state- 
ment of wcrk and quality assurance development processes. 

The Naval Facilities Engineering Command has developed 
its cwn training manual which describes the specific roles 
assigned to quality assurance evaluators. This manual is 
known as MC-326.2, "Quality Assurance Evaluators Training 
Manual" and is utilized by each Engineering Field Division 
in its quality assurance training presentations. Quality 
assurance evaluators perform the following duties which are 
listed in MO-326.2; [Ref. 42] 
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a) Review plans and service contract specif icatrcns 
before contract solicitation; 

b) Assist in pre- award surveys; 

c) Attend the pre-bid and post-award conferences; 

d) Coordinate transfer of government furnished space, 
utilities, equipment and material to the contractor; 

e) Prepare quality assurance plans; 

f) Prepare surveillance schedules, perform surveillance , 
and submit reports of findings; 

g) Review all contractor schedules and advise Service 
Contract Manager of acceptability; 

h) Assist in the preparation, or directly prepare, 
government estimates for change orders; 

i) Recommend deductions for unsatisfactory work to the 
Service Contract Manager; 

j) Monitor the contractor's safety practices and report 
results; 

k) Conduct labor standards interviews as necessary; and 

l) Conduct surveillance on the contractor's accomplish- 
ment of required corrective changes. 

It should be emphasized that this list is not exhaus- 
tive. In spite of an implicit policy that quality assurance 
evaluators should be dedicated solely to surveillance func- 
tions, they may work for functional managers and perform 
several ether duties in addition to their surveillance 
responsibilities. 

When considering QA2 responsibilities, it is evident 
that a well organized training course addressing several 
elements of contract administration is essential. 
Additionally, the course must be presented in a clear and 
concise manner using terminology that is easily understan- 
dable by the layman . Training in basic contract law, 
administration principles, cost and price analysis, organi- 
zation structure, and contract specifications should be 
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covered . Ac equally important topic which must be included 
in any quality assurance courses for service contracts is 
statistical guality assurance. As discussed earlier, the 
use cf statistical sampling techniques has been advocated 
for the inspection of commercial activities service 
contracts. It offers significant advantages over the use of 
one hundred percent inspection and reliance upon customer 
complaints. 

Specific training in the use of Military Standard 105D 
and random number tables is required. Inspectors should know 
hew tc form homogenous lot and sample sizes, establish real- 
istic acceptable quality levels ( AQL * s) , create monthly 
random sampling schedules, and conduct inspections. Finally, 
inspectors will require instruction in properly disposing of 
unacceptable variances from required performance. 
Statistical quality assurance may initially appear 
confusing, but a well designed training program will reveal 
it to be a much simplified and useful approach. Random 
sampling will be more effective than 100 percent inspection 
cr planned sampling techniques and requires fewer inspec- 
tors. A key requirement for effective service contract 
courses is the presentation cf random sampling inspection in 
a simplified and understandable manner. Different COD 
training courses have been designed with all of these 
requirements in mind; the implementation of a uniform DOD 
correspondence course could further facilitate successful 
QAE training. 

One essential requirement for an adequate training 
program is sufficient funding to cover costs of providing 
necessary training tc new quality assurance evaluators at 
field activities. Effective quality assurance programs to 
administer CA service contracts will be created and 
sustained only when activities and claimants identify, plan 
and budget for comprehensive training programs. 
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Study observations revealed that several Navy activities 
have attempted to send Quality Assurance Evaluators to 
appropriate training courses but insufficient funds have 
prevented attainment of this goal. It was observed that 
on-site training programs were generally non-existent, and 
should be initiated to enhance the skills and capabilities 
of Navy Quality Assurance Evaluators. 

D. INTEGRATING RESOURCES TO BUILD A QA PROGfiAH 

Thus far, three basic building blocks for an effective 
guality assurance organization have been discussed. These 
are the obtaining of sufficient personnel, training of 
QAE’s, and funding of inspection resources. If any cf these 
elements are inadequate, the quality assurance program to 
support A-76 contract conversions will be jeopardized. 
Perhaps the underlying intangible element that ties all of 
these together is commitment. The activity Commanding 
Officer, the base personnel, the major claimants, and the 
various contracting agencies will need to form a coalition 
to accurately plan ard provide for the proper integration of 
these building blocks. In the case of the Navy, cooperative 
attempts are being pursued to bring about successful 
contracting cut . 

The authors found wide variations in these efforts. 
Formal design and implementation of quality assurance plans 
has lagged the awards of many CA contracts and many contract 
inspection systems at Navy activities are not yet formal- 
ized. Random sampling inspection is used at very few Navy 
activities that administer service contracts. Seme reasons 
for its infrequent usage are that it seems to be too compli- 
cated and that extrapolated deductions are not yet allowed 
for Navy contracts utilizing random sampling. These Navy 
activities visited believed that the formation cf a field 
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contracting system fcr service contracts that approximates 
the organization for construction contracts will make the 
administration of such contracts more viable and effective. 

The size of the inspection organization and the sophist- 
ication cf the predesigned quality assurance plan determine 
the adequacy of the overall contract administration system 
for CA contracts. Adequate staffing and a sound inspection 
plan will lead to successful surveillance, winning the coop- 
eration and support of both the contractor and the base 
communities that receive services. This plan will optimize 
the use of inspection resources. Effective scheduling of 
inspectors and audits of contract administration can be 
achieved. An alternative, less optimal approach is to allow 
quality assurance planning to be limited by inspection 
resources that are available to the activity. It is likely 
that this option will be the one most often chosen if activ- 
ities do net perform the necessary pre-contract planning and 
budgeting fcr the formulation of CA contract organizations. 



77 



VI. ORGANIZATIONS FOR A- 76 SERVIC E CONTRACT MANAGEMENT 

A. MANAGEMENT OP SERVICE CONTRACTS AT NAVFAC HEADQUARTERS 

Preceding discussions have sen the tone for the predomi- 
nant issue of this chapter, the makeup ana location of 
service contract inspection and administration organiza- 
tions. It is first appropriate to examine the organizational 
structure for management of service contracts at NAVFAC 
headguarters . 

NAVFAC has located management of service contract policy 
for facility support contracts with the Assistant Commander 
for Maintenance and Transportation (code 10) . This is also 
true for the management of service contracts at the 

Engineering Field Division level, where a similar 
Maintenance Division (code 10) oversees the evolution of 
Navy CA service contracting. Authority for all contracts 
(Commercial Activities or construction) is vested in the 
Acquisition Department (code 0 9 A ) at both headquarters and 

EFD levels. A typical Engineering Field Division performs 
the following management and contract duties for A-76 
procurement: [Ref. 43] 

a) Distributes contracting directives and provides 
guidance to field contract offices; 

b) assists activity contract offices during contract 
dis cutes ; 

c) oversees the operation of activity contract offices to 

ensure the integrity of CA service contract 

processes ; 

d) reviews activity requests for sole source procurements 
and refers these to NAVFAC when appropriate; 
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I r> late 1982, a few cf NAVFAC’s Engineering Field 
Divisions began to consider the consolidation cf all 
contractual, planning, training and technical support for CA 
service contracts in one Facility Support Contract Branch. 
This head cf this branch nay be delegated contractual 
authority that is separate and distinct from that authority 
held by the department head for facilities construction and 
may be able to further delegate CA contract authority to 
responsible cognizant individuals at the field level (such 
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as the field activity’s Public Works Officer) . This concept 
is still under study, offering striking possibilities for 
the future administration of NAVFAC's service contracts. 
Perhaps NAVFAC will consider the creation of a separate 
service contract organization that is an exact likeness of 
the one for construction since the staff and expertise to 
manage such a program is largely in existence at N AV FAC 
headguarters , Engineering Field Divisions, and the large 
Public Wcrks Centers. Such a new organization would require 
the establishment of field activity service contract offices 
to prcmote uniformity in administration of A-76 contracts 
and establish management control of the service contract 
process, and in turn promote NAVFAC's credibility as a 
service contract management agent. 

B. THE ACTIVITY ORGANIZATION STRUCTURE 

1 • Introduc tion 

In the current A-76 setting, most commercial 
functions are related to facilities or supply management. 
Accordingly, the officials who are responsible for these 
areas at most naval activities are either the Public Works 
Officer or the Supply Officer. Such functions are vital to 
successful accomplishment of the activity mission, and 
failure to provide them in a timely, cost effective manner 
could seriously jeopardize the operating posture of the 
activity. Most functions are currently performed by federal 
civilians cr military personnel, but growing trends in 
contracting cut under the A-76 policy signal a change. 

2 . Typical Public Works De partm ent Fu net ion s 

A sample of the different roles performed at a 
typical Navy Public Wcrks Department is as follows: 
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a) Engineering Division 

i) Prepares and reviews data for military ccnstr 
tion and special projects programs; 

ii) Provides design services and prepares plans and 
specifications; 

iii) Interfaces with architect-engineer contractors; 
and 
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ii) Arranges inspections and maintenance of family 
housing; and 

iii) Assists in budget preparations. 

d) Maintenance Division 

i) Maintains all facilities; and 

ii) Accomplishes maintenance under emergency service 
requests cr specific job orders. 

e) Utilities Division 

i) Operates and maintains all command utilities 
equipment and structures. 

f) Transportation Division 

i) Operates and maintains a motor pool; 

ii) Examines and licenses all government equipment 
operators; and 
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iii) Identifies and determines transportation equip- 
ment requirements. 

This breakout of Public Works Department tasks idsn- 
xifies areas subject to CA cost studies: maintenance, util- 

ities, and transportation operations. At many Navy 
activities, such cost studies are in progress. 

3 • £ - 3 v io u s N A V F AC CA Servic e Co ntrac t A dmin istratic n 

An important step in the A-76 service contract 
process is the choice cf a suitable organizational structure 
to manage the activity's contract administration responsi- 
bilities. NAVFAC has left the choice of the size, location, 
and professional skill capabilities for this new organiza- 
tion to the discretion of the activity Commanding Officer. 
He is tasked to provide surveillance since contracting 
agents have not been given sufficient funds or personnel 
resources to manage CA contracts. 3efore the Commanding 
Officer builds such an organization, he should be informed 
of these factors which should influence its design. 

Traditional work planning, control and pest perfor- 
mance inspection of Public Works maintenance and service 
operations have been performed by the Maintenance Control 
Division (KCD) of the Public Works Department (possibly 
excluding utilities and transportation operations). MCD 

inspectors prepare job orders for shops personnel and esti- 
mate time and material requirements based upon engineered 
performance standards. Inspectors periodically visit job 
sites to verify proper performance of repairs and mainte- 
nance. They also compare actual job expenditures with orig- 
inal estimates. As CA service contracting became part of 
Navy facilities management, inspection responsibilities have 
been assigned to Maintenance Control Division personnel. 
Initial types cf contracts inspected include custodial 
services, grounds maintenance, refuse collection, and 
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housing facilities maintenance. Sometimes this responsi- 
bility is shared between MCD inspectors and base tenants 
when the volume of contracted services increases signifi- 
cantly. This shared responsibility has resulted in somewhat 
tenous CA ccntract administration. 

Eeliance upon base customers to provide required 
surveillance is not always a suitable practice. There is 
always a danger that customers acting in a contract inspec- 
tor's rcle will exercise implied or apparent authority in 
dealing with the contractor. Constructive changes cr unau- 
thorized changes in scope can lead to counterproductive 
disputes and litigation; these dc not serve the best inter- 
ests of either the government or the contractor. Where 
customer surveillance methods are used in a way that 
ccntract performance standards are ultimately raised, then 
relief will generally be provided to the contractor under a 
changes cr disputes process. A surveillance system which 
depends heavily upon customer observation of a contractor is 
not desirable for use in the CA service ccntract quality 
assurance process. [Bef. 42] 

Increasing wcrk volumes attributable to service 
contract conversions make establishment of centralized 
service contract management organizations advantageous to 
successful CA contracting. Such organizations are separate, 
distinct station divisions, act as key monitors of service 
contractor activity, and will probably be located within 
Public Works or Supply Departments. The CA contract inspec- 
tors in such divisions possess a wide range of technical 
skills sc that full contract surveillance can be provided. 
This arrangement enhances management control and maintains 
integrity in the contract process. 

If a centralized organization (with all QAI's 
working under one quality assurance manager) is net attai- 
nable, then other less optimal organization arrangements 
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exist. These are illustrated and explained in Figures G. 1 
through G.4 in Appendix G. These figures show that a 
Facility Support Contract Office can administer CA service 
contracts of a facilities nature, being located as a sepa- 
rate division of the Public Works Department (PWD) or as a 
subdivision of the PWD Maintenance Control Division. QAE's 
may work in the proposed central organization, or may work 
for functional division managers. [Ref. 45] 

4 . The Nay v Serv ice Contract 3a n ag er 

A key player in NAVFAC's service contract process is 
the Service Contract Manager ( SC 3 ) , the head representative 
of the activity's guality assurance team. His role is 
described as follows: [Ref. 28] 



The Service Contract Manager is that person with direct 
responsibility for day-to-day managemement of the 
service contract. Prior to award he is responsible for 
assisting in the preparation of the statement of work, 
the government estimate, and the surveillance plan. 
Post award responsibilities are to ensure that the 
contract runs smoothly and is properly managed, that 
surveillance is conducted, and documented, that contract 
working files are maintained, and that work orders are 
properly coordinated with the Offrcer-in-Charge. If 
chance orders are required, the SC3 must process them 
and ' make a recommendation to the 

Resident-Cff icer- in-Charg e to issue a change; if the 
contractor is having problems, the SCM must recommend 
required action to the Resident-Off icer- in-Charge in 
matters involving quality, time, money, or safety, and 
must coordinate these matters with tne contractor, the 
contract specialist, and the Resident-Of ficer-rn-Char ge . 

The quality assurance program provides the Service 
Contract Manager with information on the contractor's 
performance. The SCM has technical and supervisory 
respcnsibilit y for this program. 



This individual may be either an Engineering Field 
Division asset or may be on the field activity's rolls, and 
possesses an appropriate level of contract authority. The 
SCM is skilled in the use of statistical guality assurance 
and is able to plan and manage random sampling surveillance. 
He probably has a public relations role. Navy SCM's may be 
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either military officers or civilians. For the Air Force, 
this player is known as the Quality Assurance Evaluator 
Program Coordinator; the major difference between Air Force 
and Navy roles is that the SCM has supervisory responsi- 
bility ever QAE's, while the QA2 Program Coordinator poss- 
esses advisory responsibility only. 

5 . A Cho ice Of Ce ntrali zed or De ce ntralize d 

Organization s 

In creating the CA service contract management orga- 
nization, several alternatives may be considered, ranging 
from decentralized to centralized formats. The centralized 
organization has already been discussed; the Service 
Contract Manager is completely in charge of this type of 
structure. Contract specialists and quality assurance 
personnel report to the SCM who usually is a division head 
reporting to the activity Public Works Officer or Supply 
Officer . 

At the ether extreme, the SCM is a staff advisor 
with partial supervisory responsibility over inspectors, 
with all quality assurance personnel being assigned to func- 
tional department managers. Ihe SCM may find it difficult to 
mandate specific detailed procedures to be followed in 
executing the surveillance pregram, with the possible less 
of management control being most pronounced under this 
scheme. 

Other variations might locate the contract manager 
within ether divisions. For example, he may report to the 
activity's Maintenance Control Director. There may also be 
mere than one SCM manager for an activity. Finally, all 
functional division heads, given the proper amount and level 
cf training, could conceivably become Service Contract 
Managers . 
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The Air Force utilizes the following CA contract 
management structure. All Quality Assurance Svaluators are 
assigned to functional managers and are assisted in their 
duties by the QAE Program Coordinator, an expert in the 
field of service contract surveillance. QAE’s are always 
designated in writing, and a candidate's final approval is 
decided by the installation commander. Training and surveil- 
lance of QAE activities is handled by the QA program coordi- 
nator. Location of the quality assurance organization is 
decentralized pursuant to policy stated in AFH 70-9. Some 
Air Force activities formerly utilized centralized QAE orga- 
nizations; the authors' conversation with Air Force sources 
revealed that such organizations no longer exist. 

No Navy policy exists that mandates either decen- 
tralized or centralized organizations. At larger Public 
Works Centers, trends seem to indicate that centralized 
types of cr ganizations manage the administration of service 
contracts. Within Public Works Departments, either type of 
organization may be found. Eased on the small sampling 
taken, larger Public Works Departments prooably locate all 
QAE assets and the Service Contract Manager within the 
Maintenance Control Eivision. 

Centralization of QAE assets is desirable for 
several reasons. One is that effective contract management 
control must be established. Inspectors placed under func- 
tional managers may have little or no allegiance to main- 
taining the integrity of the service contract process and 
may net be able to conform with standards set by the SCM. 
There are :.c guarantees that Quality Assurance Evaluators 
will be dedicated strictly to performance of inspection. 
Thus, overall integrity of the service contract process may 
he sacrificed where the surveillance responsibility accrues 
to a functional division head. Work ordering inputs may not 
he kept distinct from work output verification processes for 
the contract, leading to possible conflicts of interest. 
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Special qualities of the Service Contract Manager 
were described earlier. His responsibility for growing 
numbers of A-76 service contracts makes it necessary to 
appoint kirn as a division head. 

The most important reason not to locate QAE 
personnel under functional management is to preclude the 
inadvertent communication of invalid implied contractual 
authority tc the service contractor. Air Force limitations 
on QAE roles are spelled out as fellows: [Ref. 38] 

a) QAE's will not clarify, interpret or infer legal 
interpretations of contract scope or intent; 

b) QAE's will not give direction to contractor employees; 

c) QAE's will not enter into unauthorized contract agree- 
ments (including modifications) ; 

d) QAE's will not require work to be done that is not 
specifically called fer in the contract; and 

e) QAE's will not authorize expenditures of funds. 

These actions are the responsibility of the cogni- 
zant contracting agent. In a centralized staffing arrange- 
ment, the SCM can readily identify and ascertain the actions 
of the Quality Assurance Evaluators via daily inspection 
reports. The SCM will be more successful in providing 
training for QAE's and assisting in their professional self 
development. The Quality Assurance Evaluators can mere 
readily avail themselves of contract administration assis- 
tance and cooperate to establish integrity in the CA 
contract process. 

Functional managers may oppose this centralized 
inspection organization concept on the grounds that they 
will lose direct control cf their operations. They may 
strongly believe that the quality assurance function is an 
inherent responsibility of their organization, and that no 
outside centralized department should interfere with their 
assigned missions. 
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One distinct benefit resulting from a centralized 
arrangement is easier tracking and budgeting for the full 
costs of service contract administration. Such costs may be 
more difficult to identify for decentralized organizations. 

An area of uncertainty in the use of a centralized 
organization involves job gualif ications for QAE's who 
inspect unigue technical CA functions. In the Public Works 
environment, this issue might pertain zo inspection of util- 
ities and transportation eguipment service contracts. The 
activity must balance its choice between requirements for 
specialized inspection skills and general knowledge of the 
technical area. 

6 . GAO Exa min es POD Managem ent Control of Se rvice 

Contracts 

Two recent General Accounting Office reports which 
examined and criticized existing federal agency administra- 
tion of facility service contracts lend credence to the 
establishment of centralized inspection organizations. 

A report entitled "Better Management Needed in POD 
to Prevent Fraudulent and Erroneous Contract Payments to 

Reduce Peal Property Maintenance Costs" of 9 January 1980 

uncovered several instances cf overpayments to contractors 
where work was either not performed or found to be unsatis- 
factory. GAO pointed out a lack cf effective inspection 
procedures and internal management controls. Inspector 
reports were often erroneous and unreliable. GAO called for 
independent audits of each inspector's performance. 

Specific GAC criticisms were: [ Ref» 46] 

a) work performed was not billed in accordance with 

contract provisions; 

b) inferior work was accepted; 
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c) 1 ass expensive materials were substituted for these 
specified in the contract; and 

d) seme work was paid for more than once. 

Seme GAO recommendations for improvements were as 
follows: [Bef. 46] 

a) ensure that activities provide sufficient numbers of 
adequately trained contract inspectors; 

b) require that routine independent tests cf each inspec- 
tor’s work be made; 

c) ensure that the proposed work is adequately planned 
before contract award and that contract specifications 
are clear and appropriate; 

d) continue to devote a portion of internal audit effort 
tc local procurement activities; and 

a) require that detailed inspection records, including 
measurements and calculations, be maintained in 
support of contract payments. 

These recommendations might be better attained with 
a centralized CA contract managmenent organization. The 
National Aeronautics and Space Administration was criticized 
by GAG for certain inadequacies in its contract management 
procedures in a 21 October 1980 report. Specific findings 
included: [Bef. 47] 

a) a contractor was working without approved work orders; 

b) questionable reimbursements occurred for the contrac- 
tor 1 s wor k ; 

c) contract funds were increased before the need was 
justified; and 

d) seme contracting officers had a general attitude that 
small dollar value contracts were not worthy of 
adequate attention. 
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Again, such problems might oe mitigated under a 
centralized mode of contract surveillance and administra- 
tion. Ecth reports emphasize the importance of properly 
organizing and locating a CA service contract administration 
organization. The activity may obtain the right number of 
inspectors, properly train them, and then gain nothing if it 
does net join these professionals as a unified team to 
handle newly acquired surveillance responsibilities. The 
Northern Division cf the Naval Facilities Engineering 
Command (NORTHDI V) is investigating alternative structural 
arrangements (depicted in Appendix G) and has asked customer 
activities tc provide opinions pertinent to this organiza- 
tional issue. [Ref. 45] As stated earlier, no mandatory 
NA7FAC policy has been established. A wide divergence in 
attitudes and motivations at various Navy activities may be 
reason tc refrain from mandating a specific policy. 
Immediate dialogue on the issue at DOD policy-making levels 
may help tc establish standardized CA inspection organiza- 
tion guidelines. 

ether advantages of centralized inspection are mere 
careful assessments cf proposed changes to current workload 
and increased flexibility to adapt to these. More effective 
inspector autonomy will allow the QA2 tc set schedule prior- 
ities and choose appropriate survenllance methods. 

In decentralized organizations, inspectors may be 
constricted in exercising such flexibility and their sched- 
ules may be based on the personal whims cf the functional 
manager. Cne primary objective in creating an organizational 
format is to grant the inspectors proper responsibility and 
control. 
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• £ Summa ray cf Ad van tages and Disadv ant ages oi 

Cen tra l i zati cn 

Eased on the preceding discussion. the comparative 
advantages and disadvantages of a centralized organization 
are presented: 

a) Advantages: 

i) More effective and reliable management control 
mechanisms can be instituted by the Service 
Contract Manager to establish contract 
integrity. 

ii) Allegiance of Quality Assurance Evaluators is 
obtained in ensuring that services are delivered 
as required by the contract. No dilution of 
inspector motivation results by placing the 
inspector under functional area managers whc may 
not appreciate the nuances of contract surveil- 
lance. 

iii) Ultimate inspection costs may be less with 
centralized organizations. 

iv) Centralized divisions may promote a greater 
sense of professionalism among Quality Assurance 
Evaluators. Statistical quality assurance tech- 
niques and contract administration procedures 
are mastered by each inspector through constant 
cooperation and interface with the SCM. 

v) The contractor* s interests are better served by 
a centralized organization; lines of authority 
are more visible and understandable. 

vi) Opportunities for mismanagement and occurrance 
cf fraud should be lessened. 

vii) By creating a centralized organization, the 
activity more readily assigns a proper priority 
to management of CA contracts and gains a more 
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credible position when it requests additional 
funding cr training support. 

viii) Eetter opportunities for employee self develop- 
ment exist. Training can be easily secured and 
managed. The SCM will be able to better assess 
the necessary requirements for each inspector's 
self development, 
b) Disadvantages 

i) Certain CA functions may require technical 
expertise that is not available in the existing 
QAE resources. By not placing the surveillance 
for these under functional manager control, work 
performance may be jeopardized. 

ii) Functional manager objections must be dealt 
with. These might be a perceived lack of 
control that result from the inability to deal 
with contractors in a face-to-face relationship. 
They could also feel that centralized inspector 
organizations will be largely insensitive to 
special concerns. The and result of such appre- 
hension may be refusal to cooperate with the 
inspection organization and the contractor. 

iii) At small naval activities, the cost of a 

centralized division may exceed the cost of 
placing each inspector under functional division 
heads. 

Earlier study discussion examined the feasibility of 
a new contract agency organization structure for the CA 
service contracts that includes Navy intermediate and field 
commands. If these activities embrace the concept of 

centralized service contract divisions, the Navy will have 
moved one step closer to its creation. 
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Use of centralized organizations will be advanta- 
geous should an activity choose to use large Base Operating 
Support type contract solicitations. The activity may also 
desire tc form guality assurance teams for specific contract 
types which exhibit a combination of both decentralized and 
centralized formats. 

C. JOB SERIES DESCRIPTORS FOR SERVICE CONTRACT MANAGERS 

This study has carefully described a process for 
building a successful contract guality assurance organiza- 
tion at the field level, emphasizing the formation of a 
separate division to manage the administration of a growing 
number of A-76 CA service contracts. This occurs after the 
activity defines, estimates, and budgets personnel require- 
ments tc carry out the performance of each contract's 
quality assurance plan. This process will be an ongoing one 
as increasing CA contracting occurs. 

During this process, the activity needs to define and 
determine skill and knowledge requirements for that person 
who may prove to be most crucial in successfully managing 
the practical implementation of CA service contracts, the 
Service Contract Manager. 

A description of the Naval Facilities Engineering 
Command requirements for this person was presented in the 
preceding section. Some additional requirements of the SCM 
are discussed. 

A quality control manager must be able to investigate 
and prepare plans to meet long range quality control needs, 
establishing realistic objectives. He audits the effective- 
ness of in-house quality control procedures and spot-checks 
delivered services tc measure quality. Ha must be able to 
analyze and interpret records resulting from a quality assu- 
rance program, and be able to quickly dispose of matters 
pertaining to defective performance. 
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sampling 



He develops standard procedures for 
inspections and for collection, tabulation, and reporting of 
results tc proper authorities. He assists in surveys of 
quality control techniques utilized by potential suppliers. 
He conducts quality assurance training courses. He maintains 
cognizance cf all current industrial quality control stan- 
dards, and may research and develop new quality assurance 
techniques. [Ref. 30] 

Host Public Works Commercial Activities fail into one cf 
four general categories. Service Contract Managers should 
have general knowledge of these categories which are: 



a) General Housekeeping Services: 

i) Custodial services; 

ii) Refuse services; and 

iii) Grounds maintenance services. 

b) Building and Maintenance Sevices: 

i) Housing maintenance services;and 

ii) Industrial facility maintenance services. 

c) Transportation Equipment Operations and Maintenance 

services; and 

d) Utilities Operations and Maintenance services. 

When functions are placed under cost study, these will 
indicate the technical and managerial skills to be required 
cf Service Contract Managers and Quality Assurance 
Evaluators. Possible job series desrrptcrs for the Service 
Contract Manager will be explored which might match tech- 
nical and managerial capabilities required for this impor- 
tant activity position. Specific job series descriptions or 
Wage Grade classifications for Quality Assurance Evaluators 
will net be evaluated in the thesis due to the diversity of 
activity requirements for these positions. 
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The primary consideration in choosing Quality Assurance 
Evaluators is determining if there will be a paramount 
requirement for actual trade or craft experience. If the 
inspection duties reguires that QAE's be able to perform the 
work, then a Wage Grade classification is required. 
Otherwise, a General Schedule job description will suffice. 

Civilian personnel offices are most qualified to deter- 
mine the classifications and grades for QAE billets. It is 
imperative that the activity carefully define inspection 
work requirements so that an accurate assessment of specific 
technical and managerial skills can be determined. These 
will be the basis of the position description that will be 
used to evaluate the qualifications and prior experience of 
a prospective QAE candidate. The Service Contract Manager 
should be involved in the initial screening of such candi- 
dates. 

When the Service Contract Manager role was first defined 
by NAVFAC, it was recommended that this position be clas- 
sifed under the GS-1102 series. Contract Specialist. This 
series primarily involves the review and control over 
contracts tc protect the government's interest based upon 
business, financial, and legal standpoints. This series is 
suitable if the activity desires greater emphais cn main- 
taining the integrity of its service contracts. It does not 
address technical knowledge requirements which may be neces- 
sary tc manage diverse types of industrial functions. 

Another job series that has been suggested is the GS-809 
series. Construction Control Inspector. Representative 
functions of this classification are listed: [Ref. 43] 

a) Reviews of plans and specifications prior to contract 
sclicita tion ; 

b) attends pre-bid and pre-construction conferences; 

c) supervises conduct of site surveys; 
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d) assists in development of specifications; 

e) interfaces with contractors; furnishes requirements 
for construction scheduling, progress reporting, 
safety measures, wage and hour law requirements; 

f) observes and identifies all stages of construction, 
and takes action to correct problems; 

g) reviews contractor inspection systems and advises of 
necessary corrections; 

h) investigates and processes change orders; and 

i) interfaces with local agencies and authorities during 
construct ion . 

The GS-809 series places significant emphasis or. tech- 
nical skill requirements, seme emphasis on knowledge of 
contracting skills, and minimal emphasis on managerial capa- 
bilities. This position has been used with success for 
supervisory construction inspection positions. It would be 
most suitable for guality assurance specialists or for 
Service Contract Managers who will be responsible for 
surveillance of maintenance construction service contracts. 

Another series is the GS-810 Facilities Engineering 
Manager. It covers duties in the areas of investigations 
and surveys, planning and design, construction, research, 
and facilities engineering management. Generally speaking, 
the GS-810 series does not reflect a substantial requirement 
for directive or supervisory control and may not be a most 
suitable choice. It also lacks a requirement for general 
knowledge of contract administration procedures. [Ref. 49] 

Another alternative is the GS-1640, series. Facilities 
Manager, which reflects a requirement for broad technical 
knowledge of operating capabilities and maintenance require- 
ments for an activitiy's various types of physical plant and 
equipment. Certain specific elements of job performance 
include maintenance program planning, financial planning and 
control, and facilities requirements planning. This posi- 
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tion * s shortcoming is that it omits a r squirament for 
general knowledge of contract procedures. [Ref. 50] 

All series discussed thus far lack a requirement for 
knowledge in one area that is important for the future 
development cf CA service contracting, knowledge of statis- 
tical quality assurance techniques. This has net been 
mentioned in the preceding job series although it is essen- 
tial if new sampling inspection systems recommended by A-76 
policy are to be implemented. 

A job series that might best integrate the areas of 
quality assurance, contract administration knowledge, ana 
limited technical and managerial expertise is the GS-1S10 
series. Quality Assurance Inspector. It addresses the 
design and administration of quality assurance systems that 
involve monitoring, controlling and maintaining quality and 
reliability for delivered goods and services. Specific job 



elements include: 

a) Review of the contractor’s performance, 

b) review and acceptance of contractor quali-y 


control 


c) 


systems, 

inspection of delivered services to verify a 


ccntrac- 


a> 


tor's stated quality cf services, 

use of random sampling techniques in service 


contract 


s ) 


sur veilla nee , 

summarization and analysis of results cf in 


specticns 


f) 


of services, and 

resolution of contractor quality problems. 





The Air Force has utilized the GS-1910 series for its 
Quality Assurance Program Coordinator positions at the field 
activity level. The coordinator conducts training of all 
functional quality assurance evaluators and also assists in 
the statement of work and .quality assurance plan development 
processes. He conducts audits of ongoing quality assurance 
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and guaiity control procedures for all Air Force field 
activity contracts. He may participate in progress meetings 
that are held with contractors. Working in conjunction with 
cognizant contract administrators as well as the functional 
area’s Quality Assurance Evaluator, he is a vital part of 
the activity's service contract program. The Navy Service 
Contract Manager would perform essentially the same roles, 
except that this role is expanded to include direct supervi- 
sion of QAE's. 

Use of the GS-1910 series is recommended as the most 
suitable position description for Service Contract Managers. 
Additional training in contract administration procedures 
may be required, but this can be easily accomodated by 
sending the SCM to a contracting school (such as the Civil 
Engineer Corps Officers School at Port Hueneme, California.) 
This position is operational in nature, and requires a 
working knowledge of various statistical quality assurance 
techniques. It is unlikely any of the previously mentioned 
job series will enable the activity to secure a manager with 
this specialized background. An industrial engineering back- 
ground for prospective SCM candidates is also recommended. 

The rest of the inspection organization can supplement 
and augment the general knowledge held by the Service 
Contract Manager. Quality Assurance Evaluators should 
possess special skills in carpentry, utility equipment, 
transpor ration vehicle equipment and other specialized skill 
requirements. Eased on preceding discussions of organiza- 
tional issues, the hiring of contract specialists may be 
necessitated. Such skill and technical knowledge require- 
ments will be influenced by the size of a naval activity, 
its complexity, and the number of service contracts that are 
being performed. It may be necessary to hire statistical 
specialists at either intermediate or top level commands so 
that cngcing review and improvement of statistical surveil- 
lance methcds can be carried cut. 
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Activities oust accurately determine the qualifications 
cf the Sevice Contract Manager, especially if they elect to 
form a centralized service contract management organization. 
The ability to successfully execute Commercial Activities 
will fce tied to the skills of the SCM and his organization. 



D. SOMMABY 

This chapter has examined several issues that relate to 
the creation of an inspection organization for these service 
contracts that result from 0M3 Circular A-76 implementation. 
The Navy field activity that receives services has the major 
responsibility for ensuring proper contract administration 
by providing on-site surveillance resources. 

The activity must fellow a sequence of actions that lead 
to the creation of an onsite inspection organization. It 
must first estimate hew many inspectors it will need, making 
plans tc hire additional inspectors or utilize existing 
personnel resources. 

It must ensure that a training program is established 
which educates the inspectors in quality assurance proce- 
dures and furthers their knowledge in contract administra- 
tion principles. It must secure funding for this effort. 

The activity must choose the organization format that 
best suits its surveillance needs. Although research 
suggests that a centralized format is preferable, the 
activity may elect tc place inspectors under functional 
managers. Zither system must offer sufficient management 
control to ensure that contractual integrity is established. 

Finally, based upen the type of contracts to be awarded, 
an activity must define knowledge and skill requirements for 
the Quality Assurance Zvaluatcrs and Service Contract 
Managers. The SCM position is vitally important, and a most 
capable individual with training in statistical quality 
control techniques is required. 
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With a proper integration of each of these activities. 
Navy activities can expect to be in control of the evolution 
and proper iiiple mentation of OMB Circular A-76. 
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VII. RECOMMENDAT IONS FOR IM PROV EMENTS IN CA CONTRACT 

CONVERSIONS 

A. IMPROVEMENTS IN STATISTICAL QUALITY ASSURANCE 

It is evident than quality assurance techniques ere 
useful and beneficial in the CA service contract process, 
promoting fairness and equity to all concerned parties when 
used properly. These -Techniques should ultimately reduce 
contract costs tc the government and contractors, and lead 
to mere reliable assessment and documentation of service 
contract quality. Statistical Quality Control (SQC) appli- 
cations will be limited only by the creativity and imagina- 
tion cf those who use them. 

Accordingly, the following recommendations for the 
quality assurance process are offered. A formal working panel 
of DOD officials should be organized to review the different 
statistical quality assurance methods that are either being 
utilized or being studied. Accurate, reliable information 
that examines the merits and weaknesses of acceptance and 
estimation sampling processes should be made available to 
this panel. Perhaps an independent, unbiased, quality 
contrcl expert should be included on this panel tc ensure 
that each member properly understands the validity and 
application cf statistical quality assurance tc CA service 
contracts. The authors believe that any confusion among 
various DOD CA management officials concerning SQC techni- 
ques can be eliminated by properly and accurately stating 
which cf these can be used in ongoing A-76 contracts. 

A cost-benefit and risk analysis may be conducted to 
examine which of the two sampling processes are mere appro- 
priate fer CA quality assurance. Fixed budgetary funding 
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amounts for CA contract administration may have significant 
influence in the determination of which standard system 
should he adopted. 

B. IMPROVEMENTS IN ADMI NIS TEEING EXTRAPOLATED DEDUCTIONS 

A uniform policy should also be adopted for the extrapo- 
lated deductions process. As was observed earlier, the Air 
Force dees net deduct for defects when these are less than 
the reject number at the specified acceptable quality level 
(AQL) . New NAVFAC policy, if adopted, would call for deduc- 
tions for all observed defects and liquidated damages set as 
a percentage of the value of each observed defective work 
item. Current Air Force and proposed NAVFAC policy wo-uld 
implement extrapolated deductions from samples to lets when 
specified AQL' s are exceeded. 

The authors believe that the NAVFAC policy should be 
adopted. It is more realistic, providing more enforcement 
power in the long range evolution of CA service contracting. 
It may also be more defensible in subsequent GAO and ether 
DOD audits of each agency's CA contract administration 
efforts. The previously mentioned DOD A-76 vorkina panel 
that studies sampling techniques should be tasked to resolve 
this divergence in administration policy. 

The authors recommend an immediate resolution of this 
issue tc allow the presentation to its contractor community 
cf one uniform DCD CA policy concerning contract deductions 
and inspection. Failure to do so will hinder optimal perfor- 
mance cf government surveillance, and possibly cause it to 
become ineffective. 
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IMPROVEMENTS IN ESTIMATING INSPECTION RESOURCES 



The first major issue that was covered in this topic 
area was the proper estimation of contract administration 
resources, particularly the determination of the required 
number cf Quality Assurance Evaluators. If a policy is 
adopted that attempts to correlate the number of required 
inspectors with the number of contracts, their dollar value, 
or the number of positions being converted, then a compre- 
hensive study should be undertaken of A-76 resource expendi- 
tures by all federal agencies and activities to facilitate 
the development of an estimating model. All facets cf CA 
service contract administration resources consumed for the 
past five to ten years will need to be carefully examined; 
such a large inclusive data base will be the most reliable 
means of building this model; however, the cost and time 
requirements for its development may be prohibitive. 

The authors recommend a much simpler and more accurate 
approach similar to that which was developed by the Atlantic 
Division, Naval Facilities Engineering Command. This model, 
when used properly, should provide a relatively unbiased and 
unconstrained estimate of resource requirements. If this 
technique is used, sincere attempts must be made by all 
federal agencies to procure and place the required inspec- 
tion resources at each field activity. A careful integra- 
tion of A-76 resource planning activities in each agency’s 
annual budget process will be required. 

If suboptimization or satisficing occur in this process, 
then the first, more detailed, model should be developed 
since it will probably be more indicative of actual inspec- 
tion requirements. 
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ENHANCEMENT OF CURRENT QAE TRAINING 



During the course of the study research, the authors 
observed different training programs for Quality Assurance 
Evaluators. In general, these were sufficient in providing 
initial exposure to rudiments of contract law, sampling 
techniques, and other skill areas required of QAE * s. These 
basic training courses should be enhanced by instituting 
more ongoing activity training programs which either reit- 
erate or augment the fundamental precepts of CA service 
contract administration and, in turn, lead to a more profes- 
sional QAE staff. 

The Air Force has addressed this need in designing its 
QAE training programs. The QA Program Coordinator maintains 
a continuing dialogue with each QAE during the performance 
periods of service contracts, with contract administrators 
also offering their advice and assistance. 

The Navy has adopted the practice of offering regional- 
ized training for QAE’s at either Engineering Field 
Divisions, large Public Works Departments, or Public Works 
Centers. QAE candidates travel to these sites and spend a 
week in training. Few ongoing training programs at the 
activity level were observed. 

The authors recommend that existing agency efforts in 
the training phase of CA implementation be continued. 
Improvements should be effected in creating more ongoing 
activity training programs, perhaps adopting a programmed 
learning type of instruction technique. Training in statis- 
tical quality assurance can be improved further, with a 
greater emphasis placed on basic statistical theory and 
application . 

Funding support for QAE training is crucial to enable 
the quality assurance programs for CA service contracts to 
be successful. Haphazard and poorly planned funding will 
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jeopardize the enrichment of skills of each QAE and passably 
detract from the successful evolution of the A-76 program. 

E. FACILITATING Cl ADMINISTRATION CONTROL THROQGH 

CENTRALIZATION 

A large portion of the study addressed the creation of 
centralized CA contract administration organizations. It is 
realized that this organizational form will not be appro- 
priate for all DOD agencies and activities given a wide 
diversity in size and mission requirements; many activities 
may he reluctant to embrace this format even where it is 
proven to be feasible and appropriate. Earlier discussion, 
however, indicated that the advantages of centralization 
outweigh the disadvantages. 

This structure also has the advantage of aaining immed- 
iate management conticl and integrity of CA contract admin- 
istration in the current developmental stages of A-76 
implementation. Congressional oversight and public opinion 
will he very critical of cost overruns and contract irregu- 
larities, especially if a large number of displaced federal 
employees voice their objections. Decentralized control is 
not guaranteed to result in such abuses, but it would invite 
their occurence more readily than will centralized inspec- 
tion crganziations. Centrali zed inspection will more readily 
facilitate coordinated inspector efforts than decentralized 
formats. The credibility of inspection efforts may be 
enhanced mere with centralized organizations. 

Additional reasons for supporting creation of central- 
ized organizations are that full costs of inspection may be 
reduced, training programs may be more effectively adminis- 
tered, and lines of communication from government represen- 
tatives to contract administrators will be made mere clear. 
The authors recommend that the centralized organization 
structure be adopted. 



105 



F. SEBVICE CONTRACT MANAGER QUALIFICATIONS 

The Service Contract Manager will be a key activity 
individual whether centralized or decentralized organiza- 
tions are chosen. He should be well versed in statistical 
sampling and have a broad general oackgrcund in contract 
administration principles, as well as having some knowledge 
of the functional areas to be contracted out. The authors 
recommend use of the GS-1910 series, as it offers the most 
versitality and flexibility in overseeing CA service 
contract admin istr at icn. Other job descriptions were exam- 
ined, and in fact, seme of these have actually been utilized 
by various Navy activities. For example, the GS-1102 series 
could be utilized despite its lack of technical knowledge 
requirements. The overrriding necessity for knowledge of SQC 
techniques and their applications should lead to uniform 
acceptance cf the GS-1910 series as a CA contract adminis- 
tration manager standard. 



G. FUTURE CA QUALITI ASSURANCE IMPROVEMENTS 
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Commercial activities subject to 033 Circular A-76. These 
commands should follow NAVFAC’s lead in executing their own 
CA programs. 

Continued pursuit of the A-76 policy will change the 
character of mission performance. DOD agencies will need to 
carefully plan for their administration of service contract 
programs to preclude deterioration in the quality of the 
services they receive. 
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APPENDIX A 

LEGISLATIVE PROVISIONS 



The following are excerpts from two Public Laws that 
establish congressional policy and recurring restrictions 
concerning the conversion of DOD In-house activities to 
cont r act performance. 



Depa rtment of D e fe nse Appro priatioa A uthorization Act, 1975 . 
Public Law 93-365, August 5, 1 974 



SEC. 502. It is the sense of Congress that the 
Department of Defense shall use the least costly fcrm of 
manpower that is consistent with military requirements 
and ether needs of the Department of Defense. 

Therefore, in developing the annual manpower authoriza- 
tion requests to the Congress and in carrying cut 
manpower policies, the Secretary of Defense shall, in 
particular, consider the advantages of converting from 
one form of manpower to another {military, civilian, or 
private contract) for the perrormance of a specified 
yob. A full justification of any conversion from one 
term of manpower xc another shall be contained in the 
annual manpower requirements report to the Congress 
required by section 138(c) (3) of title 10, United States 
Code. 



D epa rtment of Defense Autho r izat ion Act, 1981 , P ubl ic Law 
96^342, Sep tem ber 8, 198 0 

SEC. 502. 



a) No commercial or industrial type function of the 
Department of Defense that on October 1, 1980, is 

being performed by Department of Defense personnel 
may be converted to performance by a* private 
ccntractcr-- 

i) to circumvent any civilian personnel 
ceiling; or 

ii) unless the Secretary of Defense provides to 
the Congress in a timely manner — 
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. notification of any decision to study 
such commercial or industrial type func- 
tion for possible performance by a 
private contractor; 

2. a detailed summary of a comparison of 
the cost of performance of such function 
by Department of Defense personnel and 
by private contractor which demonstrates 
that, the performance of such function by 
a private contractor will result in a 
cost savings to the Government over the 
life of the contract and a certification 
that the entire cost comparison is avai- 
lable; 

3. a certification that the Government 

calculation for the cost of performance 
of such function by Department of 

Defense personnel is based on an esti- 
mate of the most efficient and cost 
effective organization for performance 
of such function by Department of 

Defense personnel; and 

4. a report, to be submitted with the 

certir ica tion reguired by subparagraph 
3. showing — tna potential economic 
effect on employees affected, and the 
potential economic effect on the local 
community and Federal Government if mere 
than 50 employees are involved, of 

contracting for performance of such 
function; the effect of contraction for 
performance of such function on' the 
military mission of such function; and 
the amount of the bid accepted for the 
performance of such function by the 
private contractor wnose bid is accepted 
and the cost of performance cf such 
function by Department of Defense 
personnel, toaether with costs and 
expenditures which the Government will 
incur because of the contract. 

b) If, after completion of the studies required for 
completion or the certification and report 
reguired by subparagraphs 3 and 4 of subsection 
ail) , a decision is made to convert to contractor 
performance, the Secretary of Defense shall notify 
Congress of such decision. 

c) Ihe Secretary of Defense shall submit a written 

report to the Congress by February 1 cf each 
fiscal year describing the extent to which commer- 
cial ana industrial type functions were performed 
by Department of Defense contractors during the 
preceding fiscal year. The Secretary shall 

include m each such report an estimate of the 
percentage of commercial and industrial type func- 
tions of the Department of Defense that will be 
performed by Department of Defense personnel, and 
the percentage cf such functions that will be 
performed by private contractors, during the 
fiscal year during which the report is submitted. 

d) This section shall take effect on October 1, 1980. 
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AP PENDIX B 

EXCERPTS FROH MIL-STD-105D AHD MIL- BND BK-5 3- 1 A 



In this appendix, information contained in both Military 
Standard 105D and Military Handbook 53- 1A is presented. The 
text of MIL-STD- 105 D is given in pages 111 through 120. Key 
issues to be considered in the use of the standard are the 
lot size, the inspection intensity, and the desired accep- 
table quality level (AQL) . A determination of these varia- 
bles results in a sampling size and accept/reject numbers. 
Operating characteristic (OC) curves for various sampling 
plans are also presented in pages 121 through 127. These 
represent the protection that is offered to both the 
contractor and the government, being a function of the 
percent cf defectives that are found in samples. 

Pages 128 through 131 are selected pages from 

MIL- HNDBK-5 3-1 A which provide more detailed explanations of 
the operating characteristic curves. Page 132 is an excerpt 
from a random number table. Finally, pages 133 and 134 are a 
listing of key activities to be followed in implementing 
MIL-STD- 105D. 
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SAMPLING PROCEDURES AND TABLES 
FOR INSPECTION BY ATTRIBUTES 

1. SCOPE 



1.1 PURPOSE. This publication estab- 
lishes sampling plans and procedures for 
inspection by attributes. When specified by 
the responsible authority, this publication 
shall be referenced in the specification, con- 
tract, inspection instructions, or other docu- 
ments and the provisions set forth herein 
shall govern. The "responsible authority" 
shall be designated in one of the above 
documents. 

1.2 APPLICATION. Sampling plans des- 
ignated in this publication are applicable, but 
not limited, to inspection of the following: 

a. End items. 

b. Components and raw materials. 

c. Operations. 

d. Materials in process. 

e. Supplies in storage. 

f. Maintenance operations. 

g. Data or records. 

h. Administrative procedures. 

These plans are intended primarily to be 
used for a continuing series of lots or batches. 



The plans may also be used for the inspection 
of isolated lots or batches, but, in this latter 
case, the user is cautioned to consult the 
operating characteristic curves to find a plan 
which will yield the desired protection (see 
11 . 6 ). 

1.3 INSPECTION. Inspection is the proc- 
ess of measuring, examining, testing, or 
otherwise comparing the unit of product (see 
1.5) with the requirements. 

1.4 INSPECTION BY ATTRIBUTES. In- 
spection by attributes is inspection whereby 
either the unit of product is classified simply 
as defective or nondefective, or the number 
of defects in the unit of product is counted, 
with resp^t to a giv^n requirement or set 
of requirements. 

1.5 UNIT CF PRODUCT. The unit of 
product is the thing inspected in order to 
determine its classification as defective or 
nondefective or to count the number of de- 
fects. It may be a single article, a pair, a set, 
a length, an area, an operation, a volume, a 
component of an end product, or the end 
product itself. The unit of product may or 
may not be the same as the unit of purchase, 
supply, production, or shipment. 
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2. CLASSIFICATION OF DEFECTS AND DEFECTIVES 



2.1 METHOD OF CLASSIFYING DEFECTS. 

A classification of defects is the enumeration 
of possible defects of the unit of product 
classified according to their seriousness. A 
defect is any nonconformance of the unit of 
product with specified requirements. Defects 
will normally be grouped into one or more 
of the following classes; however, defects 
may be grouped into other classes, or into 
subclasses within these classes. 

2.1.1 CRITICAL DEFECT. A critical de- 
fect is a defect tha. judgment and experience 
indicate is likely to result in hazardous or 
unsafe conditions for individuals using, 
maintaining, or depending upon the product; 
or a defect that judgment and experience 
indicate is likely to prevent performance of 
the tactical function of a major end item such 
as a ship, aircraft, tank, missile or space 
vehicle. NOTE: For a special provision re- 
lating to critical defects, see 6.3. 

2.1.2 MAJOR DEFECT. A major defect 
is a defect, other than critical, that is likely 
to result in failure, or to reduce materially 
the usability of the unit of product for its 
intended purpose. 



2.1.3 MINOR DEFECT. A minor defect 
is a defect that is not likely to reduce ma- 
terially the usability of the unit of product 
for its intended purpose, or is a departure 
from established standards having little bear- 
ing on the effective use or operation of the 
unit. 

2.2 METHOD OF CLASSIFYING DEFEC- 
TIVES. A defective is a unit of product which 
contains one or more defects. Defectives will 
usually be classified as follows: 

2.2.1 CRITICAL DEFECTIVE. A critical 
defective contains one or more critical de- 
fects and may also contain major and or 
minor defects. NOTE: For a special provi- 
sion relating to critical defectives, see 6.3. 

2.2.2 MAJOR DEFECTIVE. A major de- 
fective contains one or more major detects, 
and may also contain minor defects but con- 
tains no critical defect. 

2.2.3 MINOR DEFECTIVE. A minor de- 
fective contains one or more minor defects 
but contains no critical or major defect. 



3. PERCENT DEFECTIVE AND DEFECTS PER HUNDRED UNITS 



3.1 EXPRESSION OF NONCONFORM- 
ANCE. The extent of nonconformance of 
product shall be expressed either in terms 
of percent defective or in terms of defects per 
hundred units. 

3.2 PERCENT DEFECTIVE. The percent 
defective of any given quantity of units of 
product is one hunderd times the number of 
defective units of product contained therein 
divided by the total number of units of prod- 
uct, i.e.: 



Percent defective 



Number of defectives 
Number of units inspected 



100 



3.3 DEFECTS PER HUNDRED UNITS. The 

number of defects per hundred units of any 
given quantity of units of product is one 
hundred times the number of defects con- 
tained therein (one or more defects being 
possible in any unit of product) divided by 
the total number of units of product, i.e.: 

Defects per Number of defects * 

hundred units — Number of units inspected 
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4. ACCEPTABLE QUALITY LEVEL (AQL) 



4.1 USE. The AQL, together with the 
Sample’ Size Code Letter, is used for index- 
ing the sampling plans provided herein. 

4.2 DEFINITION. The AQL is the max- 
imum percent defective (or the maximum 
number of defects per hundred units) that, 
for purposes of sampling mspection, can be 
considered satisfactory as a process average 
(see 11.2). 

4.3 NOTE ON THE MEANING OF AQL. 

When a consumer designates some specific 
value of AQL for a certain defect or group 
of defects, he indicates to the supplier that 
his (the consumer’s) acceptance sampling 
plan will accept the great majority of the lots 
or batches that the supplier submits, pro- 
vided the process average level of percent 
defective (or defects per hundred units) in 
these lots or batches be no greater than the 
designated value of AQL. Thus, the AQL 
is a designated value of percent defective (or 
defects per hundred units) that the consumer 
indicates will be accepted most of the time 
by the acceptance sampling procedure to be 
used. The sampling plans provided herein 
are so arranged that the probability of ac- 
ceptance at the designated AQL value de- 
pends upon the sample size, being generally 
higher for large samples than for small ones, 
for a given AQL. The AQL alone does not 

5. SUBMISSION 

5.1 LOT OR BATCH. The term lot or 
batch shall mean “inspection lot” or “inspec- 
tion batch/* i.e., a collection of units of prod- 
uct from which a sample is to be drawn and 
inspected to determine conformance with the 
acceptability criteria, and may differ from a 
collection of units designated as a lot or batch 



describe the protection to the consumer for 
individual lots or batches but more directly 
relates to what might be expected from a 
series of lots or batches, provided the steps 
indicated in this publication are taken. It is 
necessary to refer to the operating character- 
istic curve of the plan, to determine what 
protection the consumer will have. 

4.4 LIMITATION. The designation of an 
AQL shall not imply that the supplier has 
the right to supply knowingly any defective 
unit of product. 

4.5 SPECIFYING AQLs. The AQL to be 
used will be designated in the contract or by 
the responsible authority, Different AQLs 
may be designated for groups of defects con- 
sidered collectively, or for individual defects. 
An AQL for a group of defects may be des- 
ignated in addition to AQLs for individual 
defects, or subgroups, within that group. 
AQL values of 10.0 or less may be expressed 
either in percent defective or in defects per 
hundred units; those over 10.0 shall be ex- 
pressed in defects per hundred units only. 

4.6 PREFERRED AQLs. The values of 
AQLs given in these tables are known as 
preferred AQLs. If, for any product, an AQL 
be designated other than a preferred AQL, 
these tables are not applicable. 

OF PRODUCT 

for other purposes (eg., production, ship- 
ment, etc.). 

5.2 FORMATION OF LOTS OR BATCHES. 

The product shall be assembled into identi- 
fiable lots, sublots, batches, or in such other 
manner as may be prescribed (see 5.4). Each 
lot or batch shall, as far as is practicable, 
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5. SUBMISSION OF PRODUCT (Continued) 



consist of units of product of a single type, 
grade, class, size, and composition, manu- 
factured under essentially the same condi- 
tions, and at essentially the same time. 

5.3 LOT OR BATCH SIZE. The lot or 

batch size is the number of units of product 
in a lot or batch. 

5.4 PRESENTATION OF LOTS OR 
BATCHES. The formation of the lots or 

6. ACCEPTANCE 

61 ACCEPTABILITY OF LOTS OR 
BATCHES. Acceptability of a lot or batch 
will be determined by the use of a sampling 
plan or plans associated with the designated 
AQL or AQLs. 

6.2 DEFECTIVE UNITS. The right is re- 
served to reject any unit of product found 
defective during inspection whether that 
unit of product forms part of a sample or 
not, and whether the lot or batch as a whole 
is accepted or rejected. Rejected units may 
be repaired or corrected and resubmitted for 
inspection with the approval of, and in the 
manner specified by, the responsible au- 
thority. 

6.3 SPECIAL RESERVATION FOR CRITI- 
CAL DEFECTS. The supplier may be required 
at the discretion of the responsible authority 
to inspect every unit of the lot or batch for 



batches, lot or batch size, and the manner 
in which each lot or batch is to be presented 
and identified by the supplier shall be des- 
ignated or approved by the responsible au- 
thority. As necessary, the supplier shall 
provide adequate and suitable storage space 
for each lot or batch, equipment needed for 
proper identification amd presentation, and 
personnel for all handling of product re- 
quired for drawing of samples. 

AND REJECTION 

critical defects. The right is reserved to in- 
spect every unit submitted by the supplier for 
critical defects, and to reject the lot or batch 
immediately, when a critical defect is found. 
The right is reserved also to sample, for crit- 
ical defects, every lot or batch submitted by 
the supplier and to reject any lot or batch 
if a sample drawn therefrom is found to con- 
tain one or more critical defects. 

6.4 RESUBMITTED LOTS OR BATCHES. 

Lots or batches found unacceptable shall be 
resubmitted for reinspection only after all 
units are re-examined or retested and all de- 
fective units are removed or defects cor- 
rected. The responsible authority shall deter- 
mine whether normal or tightened inspection 
shall be used, and whether reinspection shall 
include all types or classes of defects or for 
the particular types or classes of defects 
which caused initial rejection. 



7. DRAWING Of 1 SAMPLES 



7.1 SAMPLE. A sample consists of one 
or more units of product drawn from a lot or 
batch, the units of the sample being selected 
at random without regard to their quality. 
The number of units of product in the sample 
is the sample size. 



7.2 REPRESENTATIVE SAMPLING. When 
appropriate, the number of units in the sam- 
ple shall be selected in proportion to the size 
of sublots or subbatches, or parts of the lot or 
batch, identified by some rational criterion. 
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7. DRAWING OF SAMPLES (Continued) 



When representative sampling is used, the 
units from each part of the lot or batch shall 
be selected at random. 

7.3 TIME OF SAMPLING. Samples may 
be drawn after all the units comprising the 
lot or batch have been assembled, or sam- 



ples may be drawn during assembly o f the 
lot or batch. 

7.4 DOUBLE OR MULTIPLE SAMPLING. 

When double or multiple sampling is to be 
used, each sample shall be selected over the 
entire lot or batch. 



8. NORMAL, TIGHTENED AND REDUCED INSPECTION 



8.1 INITIATION OF INSPECTION. Nor- 
mal inspection will be used at the start of 
inspection unless otherwise directed by the 
responsible authority. 

8.2 CONTINUATION OF INSPECTION- 

Normal, tightened or reduced inspection 
shall continue unchanged for each class of 
defects or defectives on successive lots or 
batchs except where the switching proce- 
dures given below require change. The 
switching procedures given below require a 
change. The switching procedures shall be 
applied to each class of defects or .defectives.; 
independently. 

8.3 SWITCHING PROCEDURES. 

8.3.1 NORMAL TO TIGHTENED. When 
normal inspection is in effect, tightened in- 
spection shall be instituted when 2 out of 5 
consecutive lots or batches have been re- 
jected on original inspection (i.e., ignoring 
resubmitted lots or batches for this proce- 
dure). 

8.3.2 TIGHTENED TO NORMAL. When 
tightened inspection is in effect, normal in- 
spection shall be instituted when 5 consecu- 
tive lots or batches have been considered 
acceptable on original inspection. 

8.3.3 NORMAL TO REDUCED. When 
normal inspection is in effect, reduced inspec- 
tion shall be instituted providing that all of 
the following conditions are satisfied: 



a. The preceding 10 lots or batches (or 
more, as indicated by the note to Table VIII) 
have been on normal inspection and none 
has been rejected on original inspection; and 

b. The total number of defectives (or de- 
fects) in the samples from the preceding 10 
lots or batches (or such other number as was 
used for condition “a” above) is equal to or 
less than the applicable number given in 
T able VIII. If double or multiple sampling 
u in use, all samples inspected should be in- 
cluded, not ''first 1 ’ samples only; and 

c. Production is at a steady rate; and 

d. Reduced inspection is considered de- 
sirable by the responsible authority. 

8.3.4 REDUCED TO NORMAL. When re- 
duced inspection is in effect, normal inspec- 
tion shall be instituted if any of the following 
occur on original inspection: 

a. A lot or batch is rejected; or 

b. A lot or batch is considered acceptable 
under the procedures of 10.1.4; or 

c Production becomes irregular or de- 
layed; or 

d Other conditions warrant that normal 
inspection shall be instituted. 

8.4 DISCONTINUATION OF INSPECTION. 

In tHe evpht that 10 consecutive lots or 
batches remain on tightened inspection (or 
such other humber as may be designated by 
the responsible authority), inspection under 
the provisions of this document should be 
discontinued pending action to improve the 
quality of submitted material. 
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9. SAMPLING PLANS 



9.1 SAMPLING PLAN. A sampling plan 
indicates the number of units of product 
from each lot or batch which are to be in- 
spected (sample size or series of sample 
sizes) and the criteria for determining the 
acceptability of the lot or batch (acceptance 
and rejection numbers). 

9.2 INSPECTION LEVEL. The inspection 
level determines the relationship between 
the lot or batch size and the sample size. The 
inspection level to be used for any particular 
requirement will be prescribed by the re- 
sponsible authority. Three inspection levels: 
I, II, and III, are given in Table I for general 
uSe. Unless otherwise specified, Inspection 
Level II will be used. However, Inspection 
Level I may be specified when less discrimi- 
nation is needed, or Level III may be speci- 
fied for greater discrimination. Four addi- 
tional special levels: S-l, S-2, S-3 and S-4, 
are given in the same table and may be used 
where relatively small sample sizes are neces- 
sary and large sampling risks can or must be 
tolerated. 

NOTE: In the designation of inspection 

levels S-l to S-4, care must be exercised to 
avoid AQLs inconsistent with these inspec- 
tion levels. 

9.3 CODS LETTERS. Sample sizes are 
designated by code letters. Table I shall be 
used to find the applicable code letter for the 
particular lot or batch size and the prescribed 
inspection level. 

9.4 OBTAINING SAMPLING PLAN. The 
AQL and the code letter shall be used to ob- 



tain the sampling plan from Tables II, III or 
IV. When no sampling plan is available for a 
given combination of AQL and code letter, 
the tables direct the user to a different letter. 
The sample size to be used is given by the 
new code letter not by the original letter. If 
this procedure leads to different sample sizes 
for different classes of defects, the code letter 
corresponding to the largest sample size de- 
rived may be used for all classes of defects 
when designated or approved by the respon- 
sible authority. As an alternative to a single 
sampling plan with an acceptance number 
of 0, the plan with an acceptance number of 1 
with its correspondingly larger sample size 
for a designated AQL (where available) , may 
be used when designated or approved by the 
responsible authority. 

9.5 TYPES OF SAMPLING PLANS. Three 
types of sampling plans: Single, Double and 
Multiple, are given in Tables II, III and IV, 
respectively. When several types of plans are 
available for a given AQL and code letter, 
any one may be used. A decision as to type 
of plan, either single, double, or multiple, 
when available for a given AQL and code 
letter, will usually be based upon the com- 
parison between the administrative difficulty 
and the average sample sizes of the available 
plans. The average sample size of multiple 
plans is less than for double (except in the 
case corresponding to single acceptance num- 
ber 1) and both of these are always less than 
a single sample size. Usually th£ administra- 
tive difficulty for single sampling and the 
cost per unit of the sample are less than-for 
double or multiple. 
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10. DETERMINATION OF ACCEPTABILITY 



10.1 PERCENT DEFECTIVE INSPECTION. 

To determine acceptability of a lot or batch 
under percent defective inspection, the ap- 
plicable sampling plan shall be used in 
accordance with 10.1.1, 10.1.2, 10.1.3, 10.1.4, 
and 10.1.5. 

10.1.1 SINGLE SAMPLING PLAN. The 

number of sample units inspected shall be 
equal to the sample size given by the plan. 
If the number of defectives found in the 
sample is equal to or less than the acceptance 
number, the lot or batch shall be considered 
acceptable. If the number of defectives is 
equal to or greater than the rejection num- 
ber, the lot or batch shall be rejected. 

10.1.2 DOUBLE SAMPLING PLAN. The 

number of sample units inspected shall be 
equal to the first sample size given by the 
plan. If the number of defectives found in 
the first sample is equal to or less than the 
first acceptance number, the lot or batch 
shall be considered acceptable. If the num- 
ber of defectives found in the first sample is 
equal to or greater than the first rejection 
number, the lot or batch shall be rejected. 
If the number of defectives found in the first 
sample is between the first acceptance and 
rejection numbers, a second sample of the 
size given by the plan shall be inspected. The 



number of defectives found in the first and 
second samples shall be accumulated. If the 
cumulative number of defectives is equal to 
or less than the second acceptance number, 
the lot or batch shall be considered accept- 
able. If the cumulative number of defectives 
is equal to or greater than the second rejec- 
tion number, the lot or batch shall be rejected. 

10.1.3 MULTIPLE SAMPLE PLAN. Under 
multiple sampling, the procedure shall be 
similar to that specified in 10.1.2, except that 
the number of successive samples required 
to reach a decision may be more than two. 

10.1.4 SPECIAL PROCEDURE FOR RE- 
DUCED INSPECTION. Under reduced in- 
spection, the sampling procedure may termi- 
nate without either acceptance or rejection 
criteria having been met. In these circum- 
stances, the lot or batch will be considered 
acceptable, but normal inspection will be 
reinstated starting with the next lot or 
batch (see 8.3.4 (b)). 

10.2 DEFECTS PER HUNDRED UNITS IN- 
SPECTION. To determine the acceptability 
of a lot or batch under Defects per Hundred 
Units inspection, the procedure specified for 
Percent Defective inspection above shall be 
used, except that the word “defects” shall be 
substituted for “defectives.” 



11. SUPPLEMENTARY INFORMATION 



11.1 OPERATING CHARACTERISTIC 
CURVES. The operating characteristic curves 
for normal inspection, shown in Table X 
(pages 30-62), indicate the percentage of 
lots or batches which may be expected to be 
accepted under the various sampling plans 
for a given process quality. The curves shown 
are for single sampling; curves for double 



and multiple sampling are matched as closely 
as practicable. The O. C. curves shown for 
AQLs greater than 10.0 are based on the 
Poisson distribution and are applicable for 
defects per hundred units inspection; those 
for AQLs of 10.0 or less and sample sizes of 
80 or less are based on the binomial distri- 
bution and are applicable for percent defec- 
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11. SUPPLEMENTARY INFORMATION (Continued) 



tive inspection; those for AQLs of 10.0 or 
less and sample sizes larger then 80 are based 
on the Poisson distribution and are applica- 
ble either for defects per hundred units in- 
spection. or for percent defective inspection 
(the Poisson distribution being an adequate 
approximation to the binomial distribution 
under these conditions). Tabulated values, 
corresponding to selected values of probabil- 
ities of acceptance (P a , in percent) are given 
for each of the curves shown, and, in addi- 
tion, for tightened inspection, and for defects 
per hundred units for aQLs of 10.0 or iess 
and sample sizes of 80 or less. 

11.2 PROCESS AVERAGE. The process 
average is the average percent defective or 
average number of defects per hundred units 
(whichever is applicable) of product sub- 
mitted by the supplier for original inspec- 
tion. Original inspection is the first inspec- 
tion of a particular quantity of product as 
distinguished from the inspection of product 
which has been resubmitted after prior 
rejection. 

11.3 AVERAGE OUTGOING QUALITY 
(AOQ). The AOQ is the average quality of 
outgoing product including all accepted lots 
or batches, plus all rejected lots or batches 
after the rejected lots or batches have been 
effectively 100 percent inspected and all de- 
fectives replaced by nondefectives. 

11.4 AVERAGE OUTGOING QUALITY 
LIMIT (AOQL). The AOQL is the maximum 
of the AOQs for all possible incoming quali- 
ties for a given acceptance sampling plan. 
AOQL values are given in Table V-*A for 
each of the single sampling plans for normal 
inspection and in Table V-B for each of the 
single sampling plans for tightened inspec- 
tion. 



11.5 AVERAGE SAMPLE SIZE CURVES. 
Average sample size curves for double and 
multiple sampling are in Table IX. These 
show the average sample sizes which may be 
expected to occur under the various sampling 
plans for a given process quality. The curves 
assume no curtailment of inspection and are 
approximate to the extent that they are 
based upon the Poisson distribution, and that 
the sample sizes for double and multiple 
sampling are assumed to be 0.631n and 0.25n 
respectively, where n is the equivalent single 
sample size. 

11.6 LIMITING QUALITY PROTECTION. 

The sampling plans and associated proce- 
dures given in this publication were designed 
for use where the units of product are pro- 
duced in a continuing series of lots or batches 
over a period of time. However, if the lot 
or batch is of an isolated nature, it is desira- 
ble to limit the selection of sampling plans 
to those, associated with a designated AQL 
value, that provide not less than a specified 
limiting quality protection. Sampling plans 
for this purpose can be selected by choosing 
a Limiting Quality (LQ) and a consumer’s 
risk to be associated with it. Tables VI and 
VII give values of LQ for the commonly used 
consumer’s risks of 10 percent and 5 percent 
respectively. If a different value of con- 
sumer’s risk is required, the O.C. curves and 
their tabulated values may be used. The 
concept of LQ may also be useful in specify- 
ing the AQL and Inspection Levels for a 
series of lots or batches, thus fixing minimum 
sample size where there is some reason for 
avoiding (with more than a given consumer’s 
risk) more than a limiting proportion of de- 
fectives (or defects) in any single lot or 
batch. 



113 



TABLE / — Sample she code letters 
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TABLE !LA — Single sampling plans for normal inspection (Master table) 
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CHART H - OPERATING CHARACTERISTIC CURVES FOR SINGLE SAMPLING PUNS 

(Curvet for double tod nullipla templing are Batched at ctoaely at practicable) 




121 



TAIIli X-H-2 - SMILING PLANS FOR SAMPLE SIZE CODE LETTER: 
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Acceptance number 
Heiection number 

far (ingle aamplmg plan above (or alternative!) uae letter L). 
Acceptance not permuted at thia aample ai«e 



CHART J - OPERATING CHARACTERISTIC CURVES FOR SINGLE SAMPLINC PLANS 

(Cur»r* (or double «id multiple sampling ere matched ti closely as practicable) 
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Nolo: Binomial distribution used for portent defective computation; Poisson for defects per hundred un’ts. 
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Mole: binomial distribution used lor percent detective computation, Poisson lor delects per hundred units. 



TABLE X-G-2 - SAMPLING PLANS FOR SAMPLE SIZE CODE LETTER: 
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He|eciion Dumber. 

U*« tingle sampling j‘*4n tbo«e (or aiicrnDiivrly ute Ulter K) 
Acceptance no I pe mulled *1 ihi* fttmplc *U«. 



TABLE AND CHART C-H FOR SAMPLE SIZE CODE LETTER: H 

CHART C-H - OPERATING CHARACTERISTIC CURVES FOR NORMAL-TIGHTENED SINGLE SAMPLING SCHEMES 
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Note: Binomial distribution used for percent defective computation; Poisson for defects per hundred units. 



in 4 1 the Sampling Plan. Two factors are generally con- 

ii "h. s e Lee t ion o t l soling plan: <1> th. eon su ».r and/ 

or supplier risk factor and (2) the economic factor. The risk pat- 
tern of each sampling plan is represented by the OC curve for e 
•> The OC curve for each plan is different, a property w 

..... l, ... .»... . £ 

in sample size and acceptance number on the acceptance or rej 
of a lot! tL proper (with respect to risk) sampling plan can be 
dLariinid fro. .tidying Che OC cutv. for ..eh plan under consider- 
ation. By studying the OC curves, it is possible to compare the 
relative risks of two or more sampling plans for a given sampling 
situation . By virtue of the OC curve, sampling tables can be con- 
Itructad in which risks of incorrect decisions have been determined 
in advance making it possible to select plans which will have risk 
factors that are acceptable to both the supplier and the consumer. 

The OC curve, then, can be used for classifying sampling plans from 
the standpoint of the protection afforded to the supplier (. Q P * 

consumer (LQ plans), or both. The economic factor must be considered 
each time a sampling plan is to be selected and, of cour ] ° cCor 
more and more important as the cost of testing go P 1 test- 

becomes especially important when, because of the high cost of t 
ill .‘.“e Ae LsAe ll.ieed t o . degree which forces • oo.pro.ise 
of the risk requirements specified for the sampling plan. Another 
lUrolcl to selecting sampling plans is used by some organizations 
which handle many types of items. Instead of selecting P g 

plan on an item by item basis as the above procedure suggests a 
standard operating procedure is established whereby a particular 
very stringent sampling plan (probably acceptance number of zero 
and 7 lar ge sample size, perhaps the entire population) is designa 
to use when inspecting any quality characteristic that may be a 
critical defect, a second but less stringent sampling plan is g 

nated to use when inspecting any quality characteristic or group o 
quality characteristics that will be at worst a maj or . defect (s) , and 
a third and still less stringent sampling plan is designated to use 
when inspecting any quality characteristic or group o qua 1 y 
acteristics that will be no worse than a minor defect(s). 

10.4.2 Effects o f Changes to the Sam pling Plan on ^aOCCovve. 

A sampling plan and its associated risks are completely define y 
the lot size, sample size, and acceptance number. The lot size, 
except in the case of very small lots, has relatively little import- 
ance in most cases in determining the risks associated with any 
given sampling plan. Thus, sample sizes and acceptance numbers are 
the two important factors which influence the risk pattern of sampl S 
plans. If the risks of a tentative sampling plan are considered un 
satisfactory, the question which follows is: What changes mus e 

made to ob/aln the’desired sampling protection?" This ^n be answered 
by considering the effect on the OC curve of changes m the sampiing 
plan. To understand the effect of such changes, a more detailed study 
of the OC curve (see Figure 2) is appropriate. From examination of 
this curve it is seen that if lots to be inspected are 2 j defective, 
approximately 90% of the lots are expected to be accepted, where 
the lots submitted are 8% defective, about 10% of the lots are 
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i Tf 27 defective and 8% defective represent 
expected to be accepted the good lots will be re- 

g ° od and bad quality 1 , P time (producer ’ s risk) and bad 

jected 10% (100 yu .lu; » risk') This rejection/ 

J j 1 ,..*. in? nf the time (consumer s nsn; . - LUJ ' 3 l j 

lots accepted but 104 or tne Tf this frequency is 

changls » th. sampling plan ana -,ulr.d. 
10 - 4 - 3 *, the^C lndn“Jd 1 ln””g»r. S “ e Ihr 1 ” 

illustrace^th^effect^chac incr.a.lng sample si., has on ashing 
the 0C curve "steeper"., 

10.1.1 nn mceocance Nomba,. Plgnm ^“““ch.'Sc 

effect of changes in Che accept ' J , the acceptance nunbet 

j rl ~ - «« 

i:.*;. . , 7 «:. ,..n«r i«.i. 
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10.4.5 Simultaneous Change of Sample Size and Acceptance Number . 

If it is desired to have more accurate disposition of the lots whose 
percent defective is close to the selected quality level (the AQL 
or the LQ for example), the sample size must be increased to provide 
more discrimination. Also, the acceptance number must be selected 
which will yield the 0C curve that is properly located about the 
"desired” quality level. Thus, if the degree of discrimination of 
a given plan is considered adequate, but the probability of accepting 
a lot at a given quality level is too great (i.e. , the plan is "too 
loose”) or too small (i.e., the plan is "too tight"), proper adjust- 
ment is made by selecting the appropriate acceptance number. Usually 
in practice, if a sampling plan is desired which has certain desirable 
risk characteristics, both sample size and acceptance numbers must be 
simultaneously adjusted (See Figure 5). In order to make proper 
adjustment, however, the effect of each must be understood. 
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FIGURE 5. EFFECT OF SIHUITANEOUS CHANGE OF SAMPLE SIZE 
AND ACCEPTANCE NUMBER ON O.C. CURUE 
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FIGURE 3. EFFECT OF CHANGING SAMPLE SIZE ON OC CURUE 
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FIGURE 4. EFFECT OF CHANGING ACCEPTANCE NUMEER ON OC CURVE 
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TAHI.E A: TAB! E 0¥ KAN l)OM NUMBER (Continued) 



^ so > 

•N A CP ® 

pm a r* cm 

N lA !N S 

? B O J' 

O—svi^r 

-P a rN pn 

N (N N O A S 



3 !*l O' IN 

aT 3 O' CP 

C CM a» a» 



— A — 
-p CP a 
S' in O' 

3 — 



a a a O 3 ® 

•J »f N N N rs 
33 ^ TO O' (M 
? Ji <•( r. O J» 
-» a O' 30 at O' 



O *** 3 A 5 



CM at _ a* rN 

O' "■' S' 3 S' 

3 » — C a 

a a at a — 

a — ■** o cm 



?>l N - 

</N A. A 
PM — 

O' fN 



o o* ® o> 
3 a o «* >» 



> a 3 p>i n* v 



pn cp -M st A O' .- 
"»(NAfMN«lN? 



— a 

S' 30 

sr a 



a ® p 

pm 3 : 



A — OkOi - 

a a at so a 

X in a 3 - 

B J ^ A • 



— • a — u 
x a x : 
Mf 30 -"> - 
-O' a pn - 



I A S' 

t PN A 

• o' o 

: a o> 



A * « 

*M A S| 

a O' a 

O A* P 



' O -• > T B 33 

'BCA- 
I SO A ? A 
> pn ® n. O' 

p cm a — a? 



3 o A A — . CP r 



' a ® ® O' 
' -I SO SO 

> — pp a X 



CP a A CM S n 



i — o-ooo 
A O' CM » aT ' 
S A J. -. -3 r 



PP X J-0"'.*'Cf'-T-S 
•# N ^ > ®0 'A®®cmOP 
NN''l-3N«-3<r 



I A N N vO a - 



. o — am a r 



"i A O' 



i 3 a *» so : 



•>T — — O-N-Tjv Or 



I J< X A N C < 



35 ® A A N : 



3 a* -o 3 ~m 

x o> cm a »? 

•M ® O' ® O' 

30 O' N js i 

*3 A A ^ sT 

aT — O 30 

in.ain.t3 — 

PM a S' — ® 

O "-J 3>-f 

3 S' —> -0 in 

* — W A — I — 

> o — ^ r>» O 

I -I ? N ; 30 

'<-‘03—® 

I pn BABA 

M O — — A O' 

r c> so A ~p 

. *3 3 J ^ *3 

r -'-i vr — a 

O so 3) O A 

A A I a — . 

J> S' -o O' 3 ' 

C C — PN O' „ 



so -o 

-VI A 
»T — 
AN ® 

3 AN 

an pn 

(M pm 
SO A. 
O’ S' 



*N x A 4 30 A S 

a -t so (— -m x -r 

^ A A ^ S' A -I 

vl *n O' C A 3 rt 

» (N - A N A ? 

A A aT A — rN ^ 

-3 '13 - - -« 

IN 4 # 3 A f>- 'J 

O -P — ® — . -| A 

!N -. (N rs j, M A 



3 a A — r 
3N A *1 O -. -. 



< - ? a O' s3 3 31 r-i 



■ S3) a 33 -S3 



- O SO A f 



"< 30 A O' A ® 
A O ^ AN 



. 'I rt -. (N # ■ 



3 - O' (N A 



• O — c* — vC 3 0 



' — O' 3 A PN . 



^ at ~P a c- 

AIA3NN 

>3 AAO A 

O 30 A -3 

3>- O » A 

A A N ® A f 

N C >3 N -J ’ 

aT ”M aT aT a v 

A o TO B SO ' 

— Cm O' 30 a 0 



N 3 O aT 3 3 — 

> — a* — — a in. 

i a ® n > j - so 



B ^ IN N ■ 



i-n — so o <■ 



■ — c 

> A — 

> O -T 



■3 O' O' A I 



o -m a a* 

<-a -o i-n o c : 

sr — *■< . 

Ai ® ^ -N r«. > 



a ® A N . 

O — f“N O r 

A C -O o 

>£ O' A N { 

® a 



B S A i 
^ J* IN 0 
3 O' N » 

B 3Bf 

AN B-3' 



> rsj A 
4 rs| A 
v -T A 



• A A 3 ^3 
NA — O — 
'OCNAI— 

' S' r- 3 S' 

I -O — B <*A x 

J 3 C n ■ 

<— i sa TO : 

A 3 O' A s 



<f f 

A sfi - 

AN 3 0 

B B O A rs 

— SO O -A O' 



Ca?B-3-.An: 



-3 B B S 

BBS: 

— • -o o - 



i INI A 

' -o — 

I N B 



fl N IN O' -3 ' 

30 -3 B -I N • 
IN. T -O — — « >. 
O' AN ® A A r 

<-* — IN C : 



' O' ® O W 



i — Ai a 

• O A - 

I IN 3) I 



i o c -j 



) O' a B 
■ i— <r -vi 
i 3 — A 
> B O' - 



<3 O A O' 



I " 



•3 B A ( 

Nf — — 

A N 0O f 
M IN B - 



N 3< B IN 

^ jn — ^n , 

A O' ^ Os 

^ — aa . 

3 ■Nl (V| — I 



*> CJN ® — 9N ® 



n — <r fN 

3 — SON -3 

, N. O 



JN A A PM 



\ PM 

C <^N 

C SNI 
(*A A 
A O 

3 O 



' SO c A PM 

N A ® PM Pa 

) IN J o B 

; >3 N IN IN 

) .O PA vO A 

' -O vT — a 

' B B B B 

• >y a — bo 

r -T PM fN A 

r B> C — — 



I A o A A P 



B N l"N O 3 

| -N ^ A PI C 

. A — A a 

P^ 3 <3 A r 

SJN PM O 3 A 

O' « O' IP ■ 

N BN 71 A s 

I pp ro & - 

® 3 — — " 



‘ N N A TO 



3 ^ 3 s 

in > 3 ' 



- 9"f O 

<3 in pp a> 

-O ao 

r A O' « jn 

»®®__ 
Ji-PSO.-M — 
I "N'T O 
I A IN N - 



r PA PP fN pp O r 



■ B3 S' 

NAP 
< A 

I SJN 



A A P 
SON a 3 - 
C O r 



a 3 - 

-7 O - 

X 0N C 
*N ^ 

-c tn a 

S' pm a 



-3 N I 
A -T - 
> — - 

A A a 
O' PM < 
PA S3 - 



J — SN- 
lO'lPJ'i 
a B 7 P N 

I SO A -3 I 

- p n p i 

M — ®PM- 

0®AO' 

' -3 C - ' 

IN B B< 

AOS*'S'' 

P — -C Aj- v 

h o -3 o : 



« O' PM 
' Ap ® 
* A ^3 

J A a3 



— PM * O' pj 

NN B B B 

C -J P O -7 

OvAAPa®- 
•3 C O' - O 

B A A - A ■ 

pp a» a ® o : 

A O' IN, — PP ; 



A B 3 0* C 



) O' — -O A — . 
•o' o ■- o — : 
'a 3X-OABS- 
I A - e N N I 

' o pm O' at 3 ; 

TO a3 A ~M — : 

' A O' a 3 UN I 

; A » B O' N ( 

I PM PI PM Pa C f 



, — A >0 -> O I 



N N B A 

3 >3 ao 3 s 

— x ao o* i 

Pa A A 3 

— A PM fN 

— . PM BO A 

SO — A pm 

N A ^ m 

pm x ao a 

Pa PP ON TO 

ao a 3 pa 

P 4 S <1 

> O' PN -. O' 

3 A O' pp B5 

PPMS0 3 — 

P PN N Nl PM 

3 O' 3 PP -O 

a Pa ® A 3 

AO'—rP-O- 
3 APa<TO- 
3 O' A A AO r 
PO-PAXf 
A PA fA A PN A 

3 -3 B B C ; 

m a o' 3 C ; 

o o cp 43 . 

B -j A B ; 

ao 3 A CP A 

’ a3 N A P N 



O O' 

IN -P 

aO a 
at *m 

O' CM 



» M3 
*a3 PP 
A 43 
PN O' 
O' — 



N B> S0> — . O 



PMaO~4T0- 

aT rM 3 S' f 

ao A PM PM N. 

<= Pa o ® a 

O' aT SO N 



' PM aO S' A 

A rM -M 3 PN 

' 3 O' BO A 

• pn O' ® — 



* CO O' — 43 c ' 

) aO A3 — pn A - 

> 43 vO -P A 3 a 

; — . M3 A — -P P 



. A aO A — PP A 

NO'aTQOA- 

! 3 S' at pn — a 

■ 3 — 3 A PM 

N « CJi Pu 

PP A O' Nl A a 

vO — 3 P P ■ 

O' ao at e 



J a0 aO S' — P 



» C x a O' a : 



'O'O'Pn-^aO — — CM <3 — 



— — bo pp : 

3 O' 43 PA s 

PN PA CP — . A 



• O' 43 ' 
| PM A f 



I A PN B M B N O 



O — -O PP G CM PP 
A CM A A Pa cm 30 






O' — CM 50 

S' a» — 3 ■ 

4 P P O , 



PP O f 
3 CM p 
O' a3 ; 



r aC O' p' cp 3 s 
1 P c p pj . 



) Pa aO O' w 

' O' CM 'T 

3 — -» o' : 

M—ON-O- 

3 ao S' at p 



M A O' 

5 SO* S' 

> 3 S' S' 

' XI b: a3 



• ® AT 
I A -O 
' O' S' 



PA - M IN B 

— —.OArM — 

— — aOS'aO 

O' — O A CM 



3 O 

CP A» 



3 X 

43 A 
3 -T 
O' pp 
O A 



O 7N N TO 3 

B IA a B IN 

A ~M O' -O — 

A aO » 

PM — i— — C 



- M N UNO 3 

CP AO VO X N| fA PN 



fN ON a* CM CP u 



' a BS A 
i o -3 : 

• ?s : 

• a? at r 

nnppBBICI 



. -O O' a PP PN . 

} *M ® 3 o aT 4 

I - A — < B a P 



. n - n j p 

} 43 CP _ 43 

' at -O 3 3 



O O ~M i 
CM - .P P 
A B 43 O' aO 

3 A PM — Pa I 

3 0>®0 — 

' B B C -3 

p -a o' ^ a 

) N B B A 



43 aO 

® O' 
PM aT 
A O' 



• ® C «3 » 



» <r a o< cm 

— — i ri ® r 



(NMBO'A-AhPAa33icB ■ 



3J<JB- 

A A 43 B P 



' 43 O' S' AT s 

) — ® PP at ; 

n PA O PA 



' A A 
I — at 
4 aO — 



< N PA O 

J — . A aT 

n B B P 



. — 3 a*<0'®0'AaOt 
*3S'3'OC'T— X ; 
13 — A — AaTAO' 

'A® IN <*P — P aI P ' 



J PA ® S' ® O 3 

N — A O' CM -* p 

' sj' a ~i o a ; 

) -p -n. -p on a a 

) a7 O' O PN ® - 



N — — PM -O - 

a cp to ® a . 

' p IN S Af 

N A B aJ A 

> P B B P 

P 3 aT PN PM 

p s' pm a cp 

3 A A? aO 3 



O' CP 
A A 

a x 



70-NPMABAB3'0PNP^AaNBS0-NP-J^'«PS»; 

^ 'O .O >0 'O -O ^ 



-pmcPaT.p OAB7C-NP4UN BABO'O 

oxxxaxxxxO'O'O'S'O'O'S'O'O'S'C 



132 



SECTION 20: SEQUENCE OF OPERATIONS IN USING MIL-STD-105 



t 



A typical sequence of operations in using the sampling pro- 
cedures and tables for inspection by attributes of MIL-STD-105 
is illustrated by Table C which follows. This table assumes a 
requirement for single sampling. 



TABLE C : Sequence of Operational Steps 



Steps 



Explanation 



1, Determine lot size. 



2. Determine inspection 
level , 

3. Determine sample size 
code letter. 

4. Determine sampling plans. 

5. Establish severity of 
inspec t ion . 

6. Determine sample size 
and acceptance number. 



7 , Select sample . 



8. Inspect sample. 



9. Record inspection 
results. 



1. Lot size controlled by lot 
formation criteria contained in 
procurement documents. Otherwise, 
establish by agreement between 
responsible authority and supplier. 

2. If the item specification does 
not give the inspection level, use 
inspection level II. 

3. Found in Table I, MIL-STD-105, 
based on lot size and inspection 
level . 

4. Single sampling generally selected. 
Double or multiple sampling may be used. 

5. Normal inspection generally used at 
start of contract or production. 

6. Assuming normal inspection and given 
the specified AQL value and the sample 
size code letter, the sample size and 
acceptance number are found in Table 

1 1 - A , MIL-STD-105. 

7. The sample, consisting of the number 
of units of product as determined from 
Table II-A, MIL-STD-105, is selected at 
random from the lot. Additionally, any 
obvious defectives that have not been 
selected for the inspection sample are 
removed from the lot (but are not in- 
cluded in the sample). (See para. 14.2) 

8. The defectives (or defects) are 
counted. If this count does not exceed 
the acceptance number (Ac), the entire 
lot is accepted. If the count equals 
or exceeds the rejection number, the 
lot is rejected. 

9. Compute estimated process average 
if required by operating procedures. 
Maintain record of accept/reject deci- 
sions in order that switching rules 
may be followed. 
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S tep s 



Exp lana t ion 



10* Resubmit lot, 10. If the lot is not accepted, it 

may be resubmitted for acceptance 
inspection only after all units of 
the lot are reinspected and all 
defective units removed or reworked. 

Example 6: Obtaining a Plan . Suppose the AQL is 1.0, the inspection 

level is II and the lot size is 2,500. The first thing required is the 
sample size code letter (usually called simply the code letter, for 
short). For a lot size of 2,500 and inspection level II, Table I 
gives the code letter as K. In the appropriate master table (Table 
II-A) , it is found that the sample size for single sampling is 125, 

AQLs for normal inspection are given along the top of the table, and 
under the value 1.0 we find the numbers 3 and 4 given under the heading 
Ac Re (which stand for acceptance number and rejection number, respect- 
ively). The sampling plan required is: 

Sample size 125 

Acceptance number 3 

Rejection number 4 

Alternatively, Table X-K-2 could be used. Again the sample size 
of 125 is found; and in the column for AQL 1.0 are found the accep- 
tance and rejection numbers 3 and 4 as before. 

Example 7: Arrows in Tables II, III, and IV . Suppose the AQL is 0.40, 

the inspection level is I, and the lot size is 230. Table I gives the 
code letter as E. Using Table II-A, it is found that there is no plan 
for letter E and AQL 0.40 but a downward pointing arrow that directs 
us to letter G instead, and the required plan is: 

Sample size 32 

Acceptance number 0 

Rejection number 1 

Alternatively, the specifying of code letter E leads us, in the ex- 
tended tables, to Table X-E-2. But this page has no column for 
AQL 0.40. Instead, the symbol of and inverted triangle appears 
for AQLs less than 1.0. This triangle refers to the footnote "Use 
next subsequent sample size code letter for which acceptance and re- 
jection numbers are available." If the triangle is thought of as an 
arrowhead, it is pointing towards the edge of the page to be turned. 
This leads to letter F where again AQL 0.40 is not given, and on to 
letter G to find the same plan as before. It is very important to 
remember that if a triangle or series of triangles directs you from 
one page to another of the extended tables, or an arrow directs you 
from one row to another of the master tables, the sample size to be 
used is the one given for the new page or the new row arrived at and 
not the one given for the original page or row [9.4]. Where upward 
pointing arrows or triangles are found the meaning is similar. The 
triangles again point to the edge of the page to be turned. 
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APPENDIX C 

EXCERPTS FROM OFPP PAMPHLET NO. 4 



Two chapters from OFPP Pamplet No. 4, "A Guide for 
Writing and Administering Performance Statements of Work for 
Service Contracts", are provided in this appendix. These are 
Chapter 4, The Surveillance Plan, and Chapter 5, Doing 
Surveillance. Chapter 4 addresses quality assurance plan- 
design and illustrates the role that HIL-STD-105D plays in 
the sampling process. Chapter 5 elanorates on the adminis- 
trative procedures necessary in conducting inspection. It 
should be noted that the OFPP approach is equivalent tc Air 
Force inspection procedures given in AFR 400-28. Finally, a 
Performance Requirement Summary (PRS) form is shown. 
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CHAPTER 4 






THE SURVEILLANCE PLAN 



4-1. Basic Approach. This chapter describes the major 
contents of a surveillance plan. There are three key ideas 
that are the basis for a surveillance plan. 

a. Management By Exception. Quality assurance relates 
to the output service provided by the contractor. As pointed 
out earlier, the output service can result either from a 
contractor-developed procedure or from an government specified 
procedure. When the procedure is specified by the government, 
compliance with the procedure is the desired output service. 

(1) When the output is based on a contractor 
developed procedure, the procedures are only looked at on a 
by-exception basis; that is, satisfactory performance of the 
output service as specified in the contract normally 
indicates that the contractor is using satisfactory 
procedures. The government should be concerned only when 
services are not adequately performed. 

(2) In this case, the inspector looks beyond the 
level of services provided only to determine if the problem 
is caused by the government or the contractor. If government 
provided items to the contractor's operation (such as, 
parts, equipment, or facilities) are at fault, action must be 
taken through government channels to correct the problem. No 
action will be required of the contractor. When the problem 
is the contractor's fault, the contractor is told to take 
corrective action. 

b. Performance Indicator. The level of contractor 
provided services is monitored by checking the performance 
values in the statement of work (SOW). As described in 
chapter 2, a performance value is a feature of the service 
that can be measured by a number. For example, two 
important performance values in vehicle maintenance and 
vehicle operations are vehicle out-of-commission (VOC) rate 
and taxi response time. 

c. Problem Location. When performance values show 
that the service is not adequately performed, the QAE uses 
decision tables to locate the problem. The tables provide a 
logical sequence to find the problem cause. Basically, they 
are a set of pointers which should find the problem's source 
in a step-by-step fashion. The construction and use of 
decision tables are described in paragraph 4-4b. 
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Figure 3-1. Performance Requirements Summary. 



4-2.* Surveillance Information Sources. There are four 
principal sources of information for surveillance: management 
information systems, random sampling, checklists, and formal 
customer complaints. The following sections describe the 
information sources in detail. 

a. Management Information Systems. In a few instances, 
an existing management information system (MIS) may be 
available as a means of surveillance. When a MIS is 
available, as in the case of the Air Force's vehicle integrated 
management system (VIMS) in the vehicle maintenance area, it 
can collect information on performance values which can 

be used instead of random sampling data. 

(1) Management information systems usually collect 
information for 100 percent of the activities for a 
specified period of time. This information can be compared 
to a contract standard. On the basis of this comparison, 
performance can be judged and the performance for the 
specified period accepted or rejected. 

(2) For example, the vehicle out-of-commission (VOC) 
rate is computed every month by the VIMS. A simple 
comparison of the VOC rate with the maximum acceptable VOC 

in the SOW explains a great deal about the level of 
maintenance service supporting the base vehicles and 
organizations . 

(3) By way of caution, however, one must check the 
data input into a MIS if the system is maintained by the 
contractor. If one is going to use a MIS to check the 
contractor, make sure the MIS contains reliable data. 

b. Random Sampling. The most frequently used way of 

service contract surveillance is random sampling. Services 
are sampled by the QAE to determine if the contractor's 
level of performance is acceptable. Acceptance sampling is 
done, basically, to determine a course of action: that is, 

whether to accept or reject the contractor's level of 
performance during a given period of time. If it rejects 
performance, certain actions are started. If it accepts 
performance, no action is taken. 

(1) The basis for doing random sampling is 
MIL-STD-105D, Sampling Procedures and Tables for Inspection 
by Attributes which is widely understood and used by both the 
government and contractors. It is based on the concept of an 
attribute. An attribute is a feature of a service which 
either does, or does not, match a standard (for example, a 
taxi is on time or it is not on time) . 
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(2) When sampling by attributes, a certain number 
of observations will match the standards and the remaining 
number will not match. Therefore, attribute sampling is 
useful for describing how a job is done, in terms of defects 
per hundred observations, or percent defective. Using this 
concept, sampling for a performance indicator can be 
developed by proceeding through a number of formal steps 
based on MIL-STD-105D. The use of these concepts is 
described in paragraph 4-3, Sampling Plan. 

c. Surveillance Checklists. Checklists are also used 
to check contract performance. They must be used sparingly, 
however. The use of the MIS and random sampling are preferred 
information sources. Checklists help in surveillance of 
contract requirements that happen infrequently. (For example, 
if a contractor is required to perform a service once a 
month, this service would be included on a checklist.) Any 
service that is not provided on a daily basis should be 
considered for inclusion on a checklist unless a MIS can be 
used to determine the quality of the service. 

/ 

d. Formal Customer Complaints. Even the best 
surveillance plan will not allow the QAE to check all aspects 
of the contractor's performance. 

(1) Formal customer complaints are a means of 
documenting certain kinds of service problems. The way to 
get and document customer complaints needs to be carefully 
planned by the persons checking the service contract. 

(2) Customer complaints are not truly random. They 
are seldom used to reject a service or deduct money from the 
contractor. 

(3) When random sampling is the chosen method of 
surveilance, a customer complaint cannot be used to satisfy a 
random observation. However, it can be used as further 
evidence of unsatisfactory performance if random sampling 
shows that the specific service is unsatisfactory. These 
complaints can be used to decide if action other than a 
deduction should be taken. 

(a) Getting Customer Complaints. An 
aggressive customer complaint program, once established, 
needs to be briefed to every organization that receives the 
contractor's services. An operating instruction should be 
given to each organization outlining the customer complaint 
program, the format and the content of a formal customer 
complaint, and the action which can be expected from those 
assigned to watching and managing the service contract. 
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(b) Documenting the Customer Complaint. 
Normally, each customer complaint is brought, either in 
person or by telephone, to the person checking contract 
performance. Enter information about the complaint into a 
Customer Complaint Record, similar to the sample shown in 
figure 4-1. The record contains the following information: 

_1 Date and time of complaint. 

2 Source of complaint - organization and 

individual. 

2 Nature of complaint (narrative 

description) . 

_4 Contract reference of complaint related 

services . 

2 Valid complaint (Yes or No). 

£ Date contractor informed of complaint. 

2 Action taken by contractor. 

2 Signature of the person receiving and 
validating the complaint. 



4-3. Sampling Plan. As a rule, a plan contains information 
on the acceptable quality level, lot size, sample size, and 
rejection level. It states the number of units from each lot 
to be inspected (that is, the sample size). It also states 
the criteria for determining the acceptability of the lot 
(acceptance and rejection numbers). This information is used 
to build the sampling guide which are the major products in a 
surveillance plan for a service contract. 

a. Beginning the Plan. To begin building a sampling 
plan, go to the Performance Requirements Summary developed 
durina the "Write Statement of Work" step, chapter 3, fiaure 

3-1. 



(1) This chart contains the required services, the 
standards, and acceptable quality levels. At this time 
decide how the services will be checked (what information 
source or method of surveillance will be used). 

(2) Show these decisions on the chart. For each 
service where random sampling is used, complete the steps 
described below. 
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CUSTOMER COMPLAINT RECORD 



Date and Time of Complaint: 21 Jan 1979 / : 1 005 



Source of Complaint 

Organization: 382 Somb Wing/LGC 

Individual: Capt John Murry 



Nature of Complaint: Called wrecker and it did not arrive until 
3 hours after the request. 



Contract Reference: F-5, para 5. 1.1. 2. 5 and Performance 

Requirements Summary. 



Validation: Contract requires a 1 hour response time. Complaint 

is valid. 



Date and Time Contractor Informed of Complaint: 21 Jan 79/: 1030 



Action Taken by Contractor: 

Contractor had a person out sick and did not have a back up driver. 
He has now developed a roster of back up drivers who can operate a 
wrecker. 



Received and Validated 3y: H. Smvth/QAE 



Figure 4-1. Customer Complaint Record. 
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b. Deciding on the Acceptable Quality Level (AQL). The 
AQL is the highest number of defects per hundred, highest 
percent defective or highest number of defects that can be 
allowed for any service performance indicator. There are 
only a limited number of AQLs listed in MIL-STD-105D but, in 
virtually all cases, one will be close enough to control the 
contractor's level of service. 

(1) The first step in designing a sampling plan 
under MIL-STD-105D is the selection of a realistic AQL. No 
service can be perfectly performed. The AQLs placed on the 
Performance Requirements Summary in figure 3-1, must be 
adjusted at this time. 

(2) Find the closest AQL from figure 4-2 and use it 
to replace the original AQL on the Performance Requirement 
Summary. For example, the AQL for taxi service might have 
been 5 percent. This would be changed to 4 percent or 6 . 5 
percent since 5 percent does not appear in the figure. 

c. Determining the Lot Size. To determine the sample 
size, the lot size must be known. The lot is how often the 
contractor provides the service in a period of time. 

(1) To determine the lot size, estimate (or count) 
the frequency of the service to be sampled, during the period 
it is to be sampled. Thus, if scheduled bus service 
timeliness is the service being sampled, and a sample is 
taken each month, the lot size is the number of times that 
are available during the month to observe bus timeliness. 

In this case, it would be the number of times the buses go 
around all the routes each day, multiplied by the number of 
days in each month on which the bus routes operate. 

(2) In the case of workorders, the monthly lot size 
can be estimated from historical information on file. The 
projected workload data gathered in chapter 2 is used to help 
determine lot sizes. 

d. Determining the Sample Size. Use figure 4-3 to 
identify an appropriate sample size for a given lot size. 

(1) Use the normal sample size column unless there 
is a limited number of QAEs or unless the cost of an 
inspection suggests the use of the medium or small sample 
size column. 

(2) Use the medium or small sample size, if 
inspections for a particular service are lengthy or hinder 
the contractor's ability to provide service to customers. 
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Allowable Acceptable Quality Levels 
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Figure 4-2- List of MIL-ST0-105D Acceptable Quality Levels. 
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Lot Size 


Normal Sample 
Size 


Medium Sample 
Size 


Small Sample 
Size 


2-8 


2 


2 


2 


9-15 


3 


2 


2 


16-25 


5 


3 


3 


26-50 


8 


5 


5 


51-90 


13 


5 


5 


91-150 


20 


8 


8 


151-280 


32 


13 


13 


281-500 


50 


20 


13 


501-1,200 


80 


32 


20 


1,201-3,200 


125 


50 


32 


3,201-10,000 


200 


80 


32 


10,001-35,000 


315 


125 


50 


35,001-150,000 


500 


200 


80 


150,001-500,000 


800 


315 


80 


500,000 and over 


1250 


500 


125 


Figure 4-3. Sample 


Size. 
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e. Selecting the Rejection Level. Use MIL-STD-105D 
to identify the acceptance and rejection level for the 
sample size (see figure 4-4). To use the figure, begin with 
the known values for the AQL and the sample size. 

(1) Find the selected sample size (in the sample 
size column) and read across that line to the column for the 
selected AQL. At that point there will either be two numbers 
or an arrow pointing up or down. 

(2) If there is an arrow, follow the direction of 
the arrow until it leads to a pair of numbers. Of the two 
numbers at the intersection or at the end of the arrow, the 
number on the left (Ac or accept) indicates the maximum 
number of defects which can occur in a sample and still 
permit the total group or lot to be judged acceptable. 

(3) When there is no accept of reject number for a 
given sample size and AQL, following the arrow will also 
cause a change in sample size. For example, with an AQL of 
1.5 and a sample size of 20, the sample size would become 32. 

(4) The number on the right (Re or reject) 
indicates the minimum number of defects that occur in a 
sample which causes the total group or lot to be judged 
unacceptable. For example, suppose the sample size is 
determined to be 32 and the AQL has been set at 6.5 defects 
per hundred. Find the number 32 in the sample size column 
and read across that line until the AQL column for 6.5 has 
been reached. The two numbers at that intersection are 5 and 
6 . 



(a) In other words, the number on the left, 5, 
is the number of defects which can be found in a sample and 
still permit acceptance of the lot. 

(b) The number 6, to the right of 5, is the 
smallest number of defects needed to declare the lot 
unacceptable and subject to further check, using the 
decision tables. 

4-4. Developing the Sampling Method. The final thing to be 
decided in sampling is how the sample will be drawn. The 
objective in the method is to insure that the sample is 
random (that is, that all services have an equal chance of 
being selected). To achieve random selection, use a random 
number table, as explained in the following examples (see 
attachment 1 for the whole table) . Most items will fall 
into one of these examples. 
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Single sampling plans for normal inspection 
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Figure 4-4. MIL-STD-1050 Acceptance, Rejection Levels for Normal Inspection. 



(3) Go to the next number, 60756. The last part of 
this number, 756, falls within the brackets one is looking 
for, so workorder 756 is selected to be sampled. The next 
random number group is 92144. Since 144 is not within the 
brackets, move to the next group 49442. Again, 442 is not 
within the brackets and therefore is not selected to be 
sampled . 



(4) This process would be continued until three 
workorders are selected. 

c. How To Use The Random Number Table To Identify 
Random Sample From a List. If a number of items need to be 
sampled that are not consecutively numbered, the simplest 
solution is to list the identifiers, for all the items in the 
lot, in a column, on a piece of lined paper. 

(1) Next, number the lines consecutively, beginning 
with the number one hundred. Now use the random number table 
to draw the sample from the line numbers. A selected line 
number leads to the identifier located on that line and that 
identifier tells which item to sample. For example, if one 
chooses to sample a set of workorders with attached sales 
slips, one is not going to have to have a set of 
consecutively numbered workorders because not every workorder 
has a sales slip attached. 

(2) List the workorders with sales slips in a 
column, number each line in the column, and randomly select 
enough line numbers to make up the sample. 

d. How To Use The Random Number Table To Identify a 
Random Sample of Days. Suppose one wants to identify 4 days 
in the month on which to sample something. The days of the 
month can be numbered 01 to 31 (or less, as appropriate). 
Begin in the random number table in figure 4-5 at 77452. 

(1) It is best to use a starting point different 
from the one used in the previous example but for the purpose 
of this example it is being used again. 

(2) One can move down the column from random number 
group to random number until the first number between 01 and 
31 is spotted. In this case, it is 23216 or, using the rule 
to discard the numbers to the left of the number of digits, 
simply 16. Thus the 16th day of the month is selected for 
sampling . 



(3) Continuing in this fashion, one discovers that 
58731, or simply 31, or the 31st, is the next day for 
sampling. Proceed in this manner until' the four days for 
sampling have been identified. 
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(4) If it is not desirable to sample on weekends, 
discard those days selected that happen to fall on a weekend 
and continue that selection until the proper number of days 
has been selected. 

e. How To Use the Random Number Table To Identify a 
Random Sample of Times of Day. If one wants to select random 
times of day to sample a service such as taxi or bus service, 
use the 24 hour clock. 

(1) If there are any constraints during each 
24-hour period, take them into consideration. For example, 
suppose that base bus service operates between 0700 and 2345. 
In this case, go through the number table until one finds a 
group of four numbers that correspond to an acceptable time 
between 0700 and 2345. Again, using figure 4-5, and 
proceeding across the line from the initial number, one comes 
to 60756 or 0756 hrs as the first random time. 

(2) The next random number is 92144 or 2144 hrs. 

The number is good and so one schedules an observation for 
2144 hrs. 



(3) Proceed in this manner until the desired number 
of sample times have been identified. 

f. How To Insure Variety in the Use of the Random 
Number Table. The use of variety in the random number table 
ensures that detectable patterns do not occur. 

(1) Besides starting at different random points and 
alternating the patterns for finding a string of random 
numbers, the user may, at some point in time, wish to use the 
first significant digits instead of the last. 

(2) For instance, in the random number group 77452 
one has customarily used the last three digits (that is, 452) 
when looking for a random number with three digits. But 
there is no reason why one could not for a period of time use 
the first three digits, or 774. 

(3) Success in using the tables requires 
consistency but also variety. The above information should 
ensure that the tables are properly used and that the sample 
is randomly drawn. 

4-5. Surveillance Plan Products. Several written documents 
are included in a surveillance plan: 

a. Sampling Guides. A sampling guide is used for 
surveillance. It is used in a surveillance plan to present 
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a. Use Of The Random Number Table. The random numbers 
in figure 4-5 are arranged in groups of five numbers (51259, 
77452, and so on) . 

(1) To use the table, begin by picking at random a 
group of numbers on any page of the table. This is usually 
done by closing the eyes and > pointing with a pencil or 
finger to some initial group/ 

(2) To identify additional random numbers, follow 
a pattern. Go along a given line to its end and then along 
the next line to its end and so on through the table until 
enough numbers have been selected or until the table ends. 

(3) If the table ends and there are still more 
numbers to select, go back to the beginning of the table and 
continue using the same pattern. Use various patterns 
alternately; for example, use lines for one sample, use 
columns for the next sample, and use a diagonal pattern for 
the third sample. 

b. How To Use the Random Number Table To Identify a 
Random Sample of Consecutively Numbered Workorders. Suppose 
one has to identify a random sample of 3 workorders for 
inspection. This can be done at the beginning of the month 
(before the workorders are written) or at the end of the 
month (to select workorders already on file). 

(1) If there are, or might be, 200 workorders to 
select from, then one begins by listing the lowest workorder 
number (known or projected). This could be #001 or possibly 
#743, for example. 

(a) List the highest workorder number (known 
or projected); in this case, it could be #200 or #943. With 
these boundaries now enter the random number table to the 
first group of numbers. For this example, use workorders 
numbered #743 to #943. 

(b) If the last three digits in the first 
group of random numbers is not between 743 and 943, discard 
that group of numbers and go to the next group. 

(2) Again, using figure 4-5, if one starts at the 
initial 77452, disregard the two numbers to the left of the 
three significant digits, or in this case, 77. The 
remaining number is 452. Since this is not between 743 and 
943, go to the next group in the same line which is 16308, 
again, discard the leftmost two numbers, and the number is 
308. This is again too low. 
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Figure 4-5. How To Use A Random Number Table. 



the information needed to sample the performance of a 
particular service. Information for the sampling guide is 
developed while the sampling information is being derived for 
the sampling plan. The steps involved in developing 
sampling information are described in paragraph 4-3. As 
shown in figure 4-6/ a sampling guide has these sections: 

(1) A statement of the AQL and its meaning in 
layman's terms. 

(2) The lot size for sampling. 

(3) The sample size. 

(4) A description of the sampling procedure 
which tells how the service will be sampled. 

(5) An explanation of the inspection procedure 
which tells what will be checked during the inspection of the 
sample . 



(6) Acceptable performance criteria which 
states the acceptance and rejection levels. 

b. QAE Decision Tables. Once a problem has been 
discovered/ the inspector must turn to a decision table and 
use the information in that table to aid him in finding the 
source of the problem. The decision table lists the symptoms 
of the problem and identifies the possible sources of the 
problem. Questions are established for each potential source 
to determine the contributing factors. A decision logic 
entry is worked up for each required service. As soon as it 
is considered satisfactory/ the information is transferred to 
the decision table. An example of a decision logic entry is 
shown as part of a decision table in the sample in figure 
4-7. 



c. Checklists. There are two main uses for checklists. 

(1) Tally Checklists. Tally checklists are used to 
document all sample observations* made during a sampling 
period. Checklists may be preprinted with any format which 
contains the following information: 

(a) Contract requirements - a statement of the 
service being inspected. 

(b) Date, time, entry for each observation. 

(c) Observation identifier of applicable 
workorder number, bus stop, or sales slip number, meal 
period, etc. 
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VO Sampling Guide #3 
Vehicle Condition Monitoring 

1 . Acceptable Quality Level (AQL) : 1 0 % 

In the long run there must be no more than 10 defects per- hundred 
vehicles. 

2. Lot Size : vehicles operated by the contractor. 

3. Sample Size : vehicles operated by the contractor. 

4. Sampling Procedure : 

At the beginning of the month, list the registration numbers of all 
contractor operated vehicles on a sheet of ruled paper. Beginning with 
the number 100, number the lines on the paper to correspond with the 
vehicle registration numbers. Using the random number table select line 
numbers equal to the sample size. The vehicle registration numbers on 
these lines indicate the vehicles to be sampled during the month. Schedule 
the inspections evenly over the month. 

5. Inspection Procedure : 

Inspect the vehicles using vehicle/ equipment discrepancy and 
maintenance report as a guide (see AFM 77-310, Vol II, Chap 6). Record 
defects per vehicle for each of the inspected vehicles. Any defects 
found not already noted by the contractor shall cause the observation 
to be recorded as unsatisfactory. 

6. Performance Criteria : 

a. Performance is acceptable when or less defective vehicles are 

discovered per month. 

b. Performance is unacceptable when or more defective vehicles 

are discovered during a month.. 

7. Phase-In Period: During the first two months of the contract the 

following AQL 1 s (paragraph 1) and performance criteria (paragraph 6) 

1 • » • 

' J • 



a. AQL: 15% 

b. Performance is acceptable if or fewer defects are discovered 

per month. 

c. Performance is unacceptable if or more defects are 

discovered per month. 



Figure 4-6. Sampling Guide. 
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Figure 4-7. QAE Decision Table. 



(d) Result of observation - either 
satisfactory or defective, 

(e) Any pertinent comment for an observation. 
An example of a checklist for sampling is shown in figure 

4-8. 



(2) Surveillance Activity Checklists. When a 
specific service or procedure occurs rarely or is not 
important enough to survey on a continuing basis , use a 
Surveillance Activity Checklist. This type of checklist must 
be prepared and included in the surveillance plan. An 
example of this kind of checklist is shown in figure 4-9. 

4-6. Contract Administrator's Plan. The contract 
administrator has an obligation to see how well the QAE is 
doing the job. The contract administrator must also make 
some independent checks of contractor performance, preferably 
by using the same techniques that go into the design of the 
QA surveillance plan. (This plan is completed at the same 
time as the surveillance plan.) As a minimum this plan must 
call for a quarterly review of the QAE 1 s use of sampling 
guides and an annual review of surveillance activity 
checklist items. 
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Vehicle Operations Sampling Guide #3 


Vehicle Condition Monitoring 


Registration 
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Time 
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Figure 4-3. Sample Tally Checklist. 
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SURVEILLANCE ACTIVITY CHECKLIST 
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CHAPTER 5 



DOING SURVEILLANCE 



5-1. Surveillance Methods. This chapter tells how to do 
surveillance once the plan is written. It tells how to build 
a monthly schedule, how to use the surveillance plan, and 
what to do when there is poor contractor performance. This 
chapter applies to Quality Assurance Evaluators (QAEs) and 
contract administrators. 

5-2. Building A Schedule. A surveillance plan is organized 
to facilitate use by the QAE. The QAE is responsible for 
developing a monthly schedule for activities, based on the 
surveillance plan's requirements. Complete the Quality 
Assurance Evaluator Schedule by the last workday of the 
preceding month and send a copy to the contract administrator 
and the functional area chief for their information and 
review. Each QAE builds a schedule by filling in the blocks 
on the schedule. Specific instructions for filling out the 
schedule are provided below. 

a. Quality Assurance Evaluator Schedule. An example of 
a surveillance schedule is show in figure 5-1. The left-hand 
side of the schedule divides the sheet into days of the week. 
This example shows only a 7-day schedule. The QAE must make 
up enough sheets to include each day of the month. Along the 
top of the schedule, insert the items to be checked during 
the month. Along the bottom of the schedule, indicate the 
number of observations to be made during the month (that is, 
how often a MIS is checked, how many samples will be taken, 
how often a surveillance checklist will be used). 

b. Filling In And Updating the Schedule. To fill in 
the inspection schedule, the QAE refers to the sampling guide 
for each service being monitored. The sampling guide is used 
with the random number table to determine the inspections 
(observations) to be made during the month (see chapter 4, 
paragraph 4-4). 

(1) Contract surveillance must cover all hours of 
operation. Random observations are scheduled at night, on 
weekends and holidays when services are performed during 
these periods. Areas that are monitored on a set schedule 
(for example, VIMS standards and analysis reports) are 
included in the monthly schedule. This monthly schedule 
shows where and what the QAE is monitoring at all times. 
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QAE Surveillance Schedule 
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Figure 5-1. QAE Surveillance Schedule. “For Official Use Only 



(2) Post any changes to the schedule weekly and 
send copies to the contract administrator and to the 
functional area chief. Document and explain the reasons for 
each change. Actual surveillance activity recorded on the 
surveillance checklist must be comparable to the monthly 
schedule. 



(3) As updated, one must be able to conduct a 
complete audit trail from the monthly schedule, to observing 
the QAE perform sampling, to completion of the surveillance 
checklist. 



(a) There must also be a correlation between 
contractor performance versus standards, AQLs , checklists and 
actions taken by the contract administrator. The sample in 
figure 5-1 shows the schedule for one week. The QAE 
completes the blank forms, indicating week of (Monday through 
Sunday), and enters the time, observation, and check (if 
pertaining to a checklist) , in the blocks corresponding to 
the item and day. 

(b) After it is completed and filled in, this 
form is " FOR OFFICIAL USE ONLY" and must not be shown to the 
contractor. 

5-3. Doing Surveillance. Doing surveillance involves using 
the surveillance plan called for in the monthly schedule. 

Use the following procedure to record observations and take 
action when the contractor's defects exceed the allowable 
number. 

a. Recording Observations. Monthly tally and 
surveillance checklists are used for each sampling guide 
and less frequently checked services. They are used to tally 
information on scheduled observations and defects noted. 

Each observation in the sample is recorded on the checklists, 
and the documents then become a formal government record for 
later reference. 

(1) When random sampling guides are used, the tally 
of observations and defects at the end of each month are 
compared to the acceptable number of defects appearing in the 
sampling guide. 

(2) The contractor is told each time an error is 
found during scheduled observations and asked to initial the 
observation recorded on the checklist. 

(3) Errors found in services not scheduled for 
observation should be brought to the contractor's attention 
but not used to count as a defect for determining if the AQL 
has been met. 



(4) Checks done with a surveillance activity 
checklist are likewise recorded. 

b. Potential Unsatisfactory Performance. If the 
sampling guide or surveillance activity checklist indicates 
that the number of defects is too high, the QAE goes to the 
decision table for that service indicator. 

(1) The QAE must locate the specific service that 
is unsatisfactory. The table will identify the possible 
causes of the unsatisfactory performance and list a number of 
questions which, if answered, will probably pinpoint the 
source of the problem. 

(2) The decision table helps the QAE identify the 
problem so that, among other things, a meaningful evaluation 
can be made of the contractor 1 s explanation and corrective 
action. For example, if the contract specifies a maximum 
out-of-commission rate for vehicles of 8 percent, and the 
rate was 10 percent, examination may reveal the excess was 
caused by excessive vehicle down for parts (VDP) . This could 
have been caused by the government's inability to provide 
timely parts support. 

(3) In such a situation, the contractor may not be 
at fault. If, on the other hand, the excessive VDP was 
created because the contractor ordered the parts on a routine 
priority rather than priority, it might be the contractor’s 
fault. The decision tables will assist the QAE in making 
such a determination. 

c. Documenting Unsatisfactory Performance. If 
performance in any area is judged unsatisfactory, the 
contractor is required to respond to a Contract Discrepancy 
Report (CDR). See sample in Figure 5-2. 

(1) The QAE prepares the form and sends it to the 
contracting officer, who signs and sends it to the 
contractor. 

(2) When completed and signed, the report, along 
with the tally checklist or surveillance activity checklist 
become the documentation supporting payment, nonpayment, or 
other necessary action. 

5-4. Taking Action. The QAE may check the contractor’s 
performance and document any non-compliance, but only the 
contracting officer may take formal action against the 
contractor for unsatisfactory performance. 
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CONTRACT DISCREPANCY RETORT 



I. «Om TMAC T WM*K« 

F12345-79-98765 

*. t*« »— r » « *T *«o*.» ©AZ) 

Vehicles Inc. - Hr. Travis Mr. Smyth 



O A T ■ J 



1 Sept 79 


1 Sept 79 




«tlw«MO •* tS"tM 

5 Sept 79 


*<Ti9«* C C n* Llf( 

10 Sect 79 


». Oi,<MI* .«CT on ..Othtw fP«n»i» m 4«mi|; iMM.nltr. i« . i. 40 - .n««n •( 



Reference the performance requirements summary Exhibit 12. The contract requires 
that taxi service meet a four minute response time with an acceptable quality level 
of 15i. Random observations indicate that this standard was not met. In a sample 
of cQ dispatches, -0 pickups exceeded the standard.. 



•• Contracting OJticar 




*• to. (Contracting Officer) 

f o to r a* ri % t a f Ti3» a 


M9-i (C •*'*•••—) \J 

. _ tc Tie: T J MME.CT f ICuf MlhCI • * t I*C. COnTtMUATIQH 



ImCKT Ip MCtUtav. fC»* 0U. **•*•<«*•• *r t—m Q.A. J 

During August, there was a limited number of drivers and vehicles available due to 
sickness and maintenance. I will initiate short morning coordination meetings each 
day at 7:30 a.m. so that maintenance and operations personnel can assure that 
enough drivers and vehicles are available for daily activities. 












The proposed corrective action and explanations are acceptable. 



« i. so • *cno**» (p.rMM Mtut, m«i*. ***•..) 



The contractor's actions should prevent further recurrence. A deduction of Sc750 will 
be made from the August Invoice computed as follows: Monthly Cost - $90,CCC 

Oeduct Percentage 1 Si 

Percent of Sample Defective 50% 

deduction So / 50 

If this recurs next month a deduction will be taken plus a cure notice will be Issued. 



eo>.TM*eroM MonriEol Mr. Travis - Manager 



j 0/£ 



i3-<_ 






JjtjCfjhZl 



CA2 



Mr. Smyth 



Mrs. Cubing 



■s£sr 
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a. Ground rules. This section lists the normal steps to 
be taken by contract administration when the QAE reports 
these deficiencies. The actions listed are not hard-and-fast 
rules , and are a minimum. More serious action can be taken 
sooner. 



(1) When the contractor's performance is 
unsatisfactory as defined in the surveillance plan and a 
formal action is indicated, the QAE, the functional area 
chief, and the contract administrator meet to determine what 
action is appropriate for the specific circumstances. 

(2) If a decision is reached not to take a monetary 
deduction, the reasons are documented. The contracting 
officer must indicate agreement with the decision by signing 
the contract discrepancy report or other decision 
documentation. 

b. Actions. Following are the actions normally taken 
when poor performance is found. 

(1) As a rule, the QAE tells the contractor's site 
manager, in person, when discrepancies occur and asks the 
contractor to correct the problem. The QAE makes a notation 
on the tally or surveillance checklist, of the date and time 
the deficiency was discovered, and has a contractor 
representative initial the entry on the checklist. 

(2) If the number of discrepancies found exceeds 
the level for satisfactory performance, the QAE uses the 
decision tables in the surveillance plan to determine the 
cause( s ) . 



(a) If the government created any of the 
discrepancies, these are not to be counted against the 
contractor' s performance. 

(b) When the government has caused the 
contractor to perform in an unsatisfactory manner, the QAE 
prepares a letter to be sent to the responsible organization 
requesting corrective action be taken. The QAE sends it to 
the organization through the contracting officer. 

(3) When the contractor is responsible for exceeding 
the limits of satisfactory performance, the contracting 
officer issues a contract discrepancy report (CDR) to the 
contractor (see paragraph 5-3c) . If the failure is serious 
enough, issue the CDR at the time of the unsatisfactory 
performance, rather than at the end of the month. 
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(4) When a CDR is issued for a specific service 
the contracting office deducts from the month's payment, an 
amount up to the percentage indicated in the Performance 
Requirement Summary exhibit of the contract. Do not delay 
the deduction until the contractor responds to the CDR. If 
surveillance was done right and the decision tables used, the 
unsatisfactory performance is clearly the fault of the 
contractor. For a specific example of a deduction, see 
paragraph 5-5. 

(5) If the contractor does not achieve satisfactory 
performance in that specific service by the end of the next 
month, the contracting officer issues another CDR and deducts 
the appropriate amount from the contractor's payment. 

(6) If a third CDR must be issued, consider issuing 
a cure notice. (However, a cure notice can be issued sooner, 
if necessary) . 

(7) Depending on the contractor's overall 
performance, the government may issue a Show Cause letter if 
the reply to a cure notice is unsatisfactory; next consider 
terminating the contract. 

5-5. Deductions For Non-Performance. Through the Inspection 
of Services clause, the government can deduct from a 
contractor's payment an amount equal to the services not 
provided . 

a. To do this, the contract administrator must know the 
major cost categories in the contract and the percentage of 
cost each service output represents. The percentage cost of 
each service is found in deduct analysis; see chapter 2, 
paragraph 2-9. An example of how the deduct formula works is 
shown in figure 5-3. 

b. Suppose the bid schedule showed the monthly contract 
price for vehicle operations, maintenance, and analysis as 
shown. The percentage cost of the service output is then 
found by looking at the Performance Requirements Summary 
Technical Exhibit in the contract statement of work. In the 
example, the percentage cost of quality of completed work is 
10 percent. This is then multiplied by $100,000 to obtain 
the maximum amount to deduct. 

c. If completed work was unsatisfactory during the 
month (that is, did not meet performance values) and the 
percent of the sample found bad was 20 percent, $2000 would 
be deducted from the payment normally due the contractor. 
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Deduct Formula (Example) 



If: 

and: 

and: 

and: 

and: 

Then: 



Figure 



Quality of completed work is unsatisfactory 
(AQL of 6.5% exceeded) 

Contract price is $100,000 per month 

Quality of completed work deduct percentage is 10% 

Sample size is 50 

Number of defects in the sample is 10 (Reject number is 8) 
Deduction from the current month's invoice is: 



Contract price = $100,000 

X Deduct percentage = .10 

510,000 

X Percent of sample defective .20 

Deduction = $ 2,000 



3. Deducting for Non-Performance. 
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d. This amount for quality of completed work is deducted 
because the contractor failed to provide reliable, uniform 
services within the assigned performance values. Although 
some completed work may have met the standard during the 
month, the acceptable quality level was not met and at least 
20 percent of the observations were defective. Hence, the 
total quality performance requirement has not been achieved. 
As a consequence, the service output is unsatisfactory. 

5-6. Good Performance. When a contractors quality control 
program works, good performance results. If the result of a 
QAE's surveillance shows consistently good performance, the 
amount of surveillance can be decreased. 

a. Reduced Inspection. Inspection can be reduced 
when the following conditions have been met for a sampling 
guide. 



(1) The preceding 4 lots (that is, the last 4 
months) have all been acceptable. 

(2) The number of defects in each of the preceding 
4 lots is less than one half of the acceptance number. For 
example, with an AQL of 6.5 percent and a sample size of 32, 
the acceptance number is 5. If two or less defects were 
found in each of the last 4 lots, reduced inspection could be 
used . 



(3) The normal sample size is being used. 

(4) The functional area chief and the contract 
administrator agree to use reduced inspection. 

b. Reduced Sample Size and Acceptance or Rejection 
Numbers. Reduced inspection decreases the sample size as 
shown in figure 5-4. In addition, the acceptance and 
rejection numbers change as shown in figure 5-5. To make the 
changes to the existing sampling guide, take the following 
s teps . 

(1) Make sure that the original sampling guide was 
using the normal sample size. To determine this, see Chapter 
4, figure 4-3 and compare the lot size with the sample size 
in the sampling guide. 

(2) Find the new sample size by using figure 5-4. 
Take the lot size and find the new reduced sample size. 

(3) Using the AQL in the sampling guide and the new 
reduced sample size, see figure 5-5 for the new acceptance 
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Lot Size 



Normal Sample Size 



Reduced Sample Size 



2-8 


2 


2 


9-15 


3 


2 


16-25 


5 


2 


26-50 


8 


3 


51-90 


13 


5 


91-150 


20 


8 


151-280 


32 • 


13 


281-500 


50 


20 


501 - 1,200 


80 


32 


1 . 201 - 3,200 


125 


50 


3 . 201 - 10,000 


200 


80 


10,001-35,000 


315 


125 


35,001-150,000 


500 


200 


150,001-500,000 


800 


315 


500,001 and over 

Figure 5-4. Reduced Sample Size. 
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Single sampling plans for reduced inspection 
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and rejection numbers. Note that there is a gap between the 
acceptance and rejection numbers (for example, sample size 32 
and AQL 6.5 percent, accept is 5 and reject is 8). This means 
that the lot would not be rejected unless 8 defects were 
found and would be accepted if 5 or less defects were found. 
However, a number of defects greater than five will be cause 
for returning to normal inspection (that is, return to the 
sample size and acceptance and rejection numbers used in the 
original sampling guide) . 

c. Returning to Normal Inspection. When reduced 
inspection is in effect return to normal inspection the next 
month under the following conditions. 

(1) When the number of defects exceeds the 
acceptance number under reduced sampling or, 

(2) The functional area chief and the contract 
administrator deem it necessary to return to normal 
inspection. 

d. Returning to Reduced Inspection. If during the 
first month of the return to normal inspection, the number of 
defects found is again less than 50 percent of the reject 
level, a return to reduced inspection may be done the next 
month. If the number of defects found is over 50 percent, 
then normal sampling must be accomplished until 4 months of 
less than 50 percent of reject level defects are found. 

5-7. Documentation. During the course of the contract the 
QAE retains a copy of all inspection schedules, tally 
checklists, and surveillance activity checklists. At the end 
of the contract period, the QAE forwards these records for 
inclusion in the contract file. However, when a specific 
service becomes unsatisfactory during a surveillance period, 
the inspection documentation supporting the contract 
discrepancy report is forwarded to the contracting officer no 
later than 5 working days after the end of the previous 
performance period . 
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AP PENDIX D 

EXCERPTS FROM NAVFAC MO-327 



This appendix includes various portions of NAVFAC MO- 327 
entitled "Service Contracts : Specifications and 

Surveillar.ee." The first illustration, the Performance 

Requirement s Summary (PSS) , is a key and comprehensive 
surveillance document. It lists each item of work to be 
performed under the contract, its required standard of 
performance, the method of inspection for the item, and the 
acceptable quality level (or allowable deviation) . 

The next series of illustrations is Chapter 5, Quality 
Assurance Methods. Each style of inspection, be it planned, 
random, etc., is explained. Criteria for choosing an appro- 
priate method is provided. Sample service inventory work- 
sheets and surveillance schedules are presented. 

The next part of this appendix is Chapter 6, 

Surveillance. Administrative responsibilities and procedures 
are listed. 

Appendix C is an abbreviated sample quality assurance 
plan for industrial solid waste collection. Appendix D is an 
instruction for the utilization of random number tables. 
Finally, current NAVFAC lot and sample tables are provided 
althcuch these will be modified by new ONR sampling tables. 
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TASK 



STANDARD & 
INDICATOR 



METHOD OF 
SURVEILLANCE 



AQL 



1. Residential waste 
collection (Sect. 
00005, Clause 3.2) 



2. Scheduled industrial 
waste collection 
(Sect. 00005,. 

Clause 3.3) 



3. Unscheduled 
industrial waste 
collection (Sect. 
00005, Clause 3,4) 

4. Ash collection 
(Sect. 00005, 
Clause 3.5) 



9. Quality control 
program (Sect. 
00004, Clause 7) 



-collection on day 
specified 
-no debris 
-cans left at curb, 
off street 

-collected within 
4 hr of specified 
time 

-no debris 

-containers properly 
repositioned 

-collected within 
12 hr of request 
for pick-up 



-collected within 
12 hr of request 
for pick-up 



-planned sampling 
(primary) 

-validated complaints 
(secondary) 



-random sampling 
(primary) 

-validated complaints 
(secondary) 



-validated complaints 



-validated complaints 



-QC program -planned sampling 

established 
-activity in QC 
files 

-QC identified 
problems resolved 



FIGURE 4-1 

Performance Requirements Summary Table 
(Sample) 



3 units 
per 100 



10 % 



5 per 
month 



3 per 
month 



zero 



CHAPTER 5: QUALITY ASSURANCE METHODS 



5. GENERAL . This section describes quality assurance evaluation 

methods used to monitor .Contractor performance on service contracts. Before 
proceeding, the purpose and goals of QA should be discussed. 

a. When the Government purchases goods or services, there must be 
some means provided to attest to the value received for monies spent. To do 
this, the Government must be able to confirm that the quantity and quality 
of goods or services received conforms to contract requirements. 

b. The recipients of the contracted goods or services, in this case 
Naval shore activities, are responsible for developing and implementing 
procedures that assure that the Government is getting what it contracted 
for. These procedures are referred to as QA. 

c. Contractors, on the other hand, are responsible for providing 
Quality Control (QC) . The purpose of QC is to control the service producing 
process, and to insure that the desired level of output quality is 
maintained. 

5.1 QA PROGRAM . 

5.1.1 Traditional Approach . The Navy's traditional approach to 
surveillance of Service Contracts, often a hit-or-miss affair with no 
written plan, has not provided adequate quality assurance. 

a. The method of surveillance which is claimed to be used most 
frequently is 100 percent inspection. In reality, however, the inspection 
is often much less than total, since 100 hundred percent inspection is very 
costly and not always feasible. 

b. Further, traditional surveillance methods have usually focused 
on the work process (adherence to specified steps and frequencies) rather 
than on the quality of contract outputs. The net result does not assure 
satisfactory quality performance. 

5.1.2 New QA Approach . The new QA approach, based on a written plan, is 
keyed to performance oriented specifications. 

a. It focuses on the quality of the product delivered by the 
Contractor and not on the steps taken or procedures used to provide that 
product. 

b. It includes appropriate use of preplanned inspections, 
validation of complaints, and unscheduled inspections. 

c. It provides a structured approach to surveillance that permits 
management control of QA. 

5.1.3 Criteria . There are several criteria for good QA. 

a. First, the PWS must be written so that the quantity and quality 
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of required work outputs are measurable. The development of the PWS and the 
QA should be viewed as a single process. These documents are interrelated; 
one defines required work outputs and quality standards while the other 
defines how work outputs will be observed and measured. 

b. QA must provide for adequate and affordable contract 
surveillance. The depth and detail of surveillance should be geared to the 
relative importance of the services monitored. 

c. QA must have the potential to support corrective action taken by 
the S CM /ROIC when non-performance or unsatisfactory performance occurs. 

d. There are three key ideas that are the basis for contract 
surveillance. 



(1) Outputs. Quality Assurance evaluates the output service 
provided by the Contractor. The output service can result either from a 
Contractor developed procedure or from a Government specified procedure. 

When the output is based on a Contractor developed procedure, the procedure 
is examined on an exceptions basis; that is, satisfactory service output as 
specified in the contract normally indicates that the Contractor is using 
satisfactory procedures. The Government should be concerned with Contractor 
procedures only when services are not adequate. When the procedure is 
specified by the Government, compliance with the procedure is the desired 
output service. 

(2) Compliance. Contractor’s compliance with contract 
requirements is monitored through the performance indicators and standards 
which are specified in the PWS. Performance indicators are measurable 
attributes of the outputs. A standard is the gauge that Contractor 
performance is compared against. For example, scheduled trash collection is 
the work required, one of the indicators of good performance is timeliness, 
and the standard is trash pick-up is made within 4 hours of the scheduled 
time. 



(3) Cause of Problem. When observed performance indicators 
show output not to be in compliance with contract requirements, the QAB 
identifies the cause of the problem. The QAE looks beyond service outputs 
to determine if the problem is caused by the Government or the Contractor. 
If the cause of the problem rests with the Government, corrective action 
must be taken through Government channels and no action is required o£ the 
Contractor. If the Contractor is at fault, the Contractor is told to take 
corrective action; payments to the Contractor are reduced; and a Contracts 
Discrepancy Report (CDR) may be issued. 

5.2 METHODS OF SURVEILLANCE . There are five methods that may be used 
for contract surveillance. 

5.2.1 One Hundred Percent Inspection . One hundred percent inspection 
requires that output from each and every work occurrence be evaluated. One 
hundred percent inspection measures the Contractor's true level of 
performance. It is an expensive and time consuming method which should be 
used sparingly. 
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5.2.2 Planned Sampling . Surveillance by planned sampling is designed to 
evaluate a part but not all of a contract requirement. 

a. The number of inspections and the items to be inspected are 
based on subjective judgment. 

b. Planned sampling is useful when requirements at one location are 
more important than those at other locations; for example, galley dumpsters 
as opposed to those in a remote administrative area. It is also useful when 
the Contractors performance is poor in some spots but better in others. 

5.2.3 Random Sampling . Surveillance based on random sampling evaluates 
part but not all of the work performed. (Appendix D provides details of the 
mechanics of random sampling.) 

a. Using random sampling, any occurrence of work is as likely to be 
monitored as any other occurrence. The QAE's bias does not affect the 
specific occurrences of work selected for evaluation since all occurrences 
of an item of work are assumed to have the same level of importance. 

b. This method estimates the Contractor's overall level of 
performance for a given contract requirement. It is most useful when 
evaluating items that are repetitive in nature such as janitorial work, 
grounds maintenance, or service call work. 

c. To achieve full benefits, the random sampling method must be 
applied properly. If misapplied, results will be biased. 

5.2.4 Validated Complaints . Validated customer complaints constitute a 
surveillance method based on customer awareness. Customers, familiar with 
contract requirements, notify the QAE when there is a case of poor or non- 
performance. Upon notification, the QAE investigates the report and, if 
valid, documents the performance problem. 

a. Formal customer complaints are a means of documenting certain 
kinds of service problems. The way to obtain and document customer 
complaints needs to be carefully planned by the persons monitoring the 
service contract. 

b. Customer complaints are not random, but when validated by the 
QAE may be used to deduct money from the Contractor. 

c. When random sampling is the chosen method of surveillance, a 
customer complaint cannot be used to satisfy a random observance. However, 
it can be used as further evidence of unsatisfactory performance if random 
sampling shows that the specific service is unsatisfactory. These 
complaints can be used to decide if other action should be taken. 

(1) Getting Customer Complaints. An aggressive customer 
complaint program, once established, needs to be explained to every 
organization that receives the Contractor's services. An operating 
instruction should be given to each organization outlining the customer 
complaint program, the format and the content of a formal customer 
complaint, and the action which can be expected from those assigned to 
watching and managing the service contract. (Appendix E provides a 
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Customers Guide for Evaluating Contractor Performance.) 

(2) Documenting the Customer Complaint. Normally, each 
customer complaint is brought, in person or by telephone, to the individual 
checking contract performance. Enter complaint information into a Customer 
Complaint Record, similar to the sample show in Figure 5-1. The record 
contains the following information: 

_1 Date and time of complaint. 

2 Source of complaint - organization or individual. 

2 Nature of complaint (narrative description) . 

2 Contract reference of complaint related services. 

5 Valid complaint (Yes or No) . 

6 Date Contractor informed of complaint. 

2 Action taken by Contractor. 

Q Signature of the person receiving and validating the 
complaint. 

5.2.5 Unscheduled Inspections . The QAE may conduct impromptu evaluations 
of contract requirements whenever necessary. This surveillance method 
(which is not really a method at all) provides no information on the 
Contractors overall performance. 

5.2.6 Criteria for Method Selection . No firm guidance for method 
selection can be provided in a manual of this type. Some general guidance 
on selection criteria are: 

a. Population size refers to the number of scheduled or expected 
occurrences of a contract requirement over a given time period, usually one 
month. The actual number of occurrences will depend on how a unit of 
service is defined. Frequency of service at any location may be daily, 
weekly, etc. Population size is easy to determine for scheduled services. 
When services are performed on an "as required" basis, population size must 
be estimated based on historical or projected data. Large homogeneous 
populations are ideally suited to random sampling. 

b. Some contract requirements are more important than others. Some 
requirements may have an impact on an activity's mission. Others have 
little or no impact. One hundred percent inspection might be used for 
"important" contract requirements, a sampling method or customer complaints 
for "less important requirements", and validated complaints for the "least 
important requirements". 

c. It is easier to talk about instances where surveillance methods 
are not appropriate. 
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(1) One-hundred percent inspection is not suited to large 
populations. It would be time consuming for the QAE and expensive to the 
Government. 



(2) Those .services that are important or costly require tight 
surveillance. Validated complaints do not guarantee that all instances of 
nonperformance or poor performance will be reported in a timely manner. 

(3) If a service is required but individual occurrences are of 
small importance (for example, emptying a trash can) , it is normally not 
beneficial to invest a great amount in surveillance. One-hundred percent 
inspection is the most costly of the evaluation methods and should not be 
used. 



(4) If a contract requirement is continuous in nature, 100 
percent inspection is not feasible since it would require a QAE to be 
on-site full time. Examples of continuous requirements are manning a guard 
post, maintaining a minimum inventory of parts, and fulfilling contractor 
quality control requirements. 

(5) It is not possible to use a surveillance method that 
requires prescheduling of evaluations for unscheduled service such as 
responding to Emergency/Service (E/S) calls, processing work orders, and 
dispatching vehicles. It is possible to schedule retrospective evaluation 
of Management Information Systems (MIS) outputs such as logs, work orders, 
or other written records. 

d. The choice between planned sampling and random sampling is 
sometimes difficult. 

(1) For example, on a Bus Service Contract it would be very 
easy to establish a random sampling evaluation plan where the QAE monitors 
the Contractor's compliance with the established bus schedule. However, 
this type of surveillance plan would require the QAE to travel to all areas 
of the bus activity on a random basis, resulting in a large amount of 
unproductive QAE travel time. 3ecause of the many customer complaints that 
would result if there were poor bus service, planned sampling is a more 
attractive surveillance method. 

(2) A good rule of thumb in choosing between random and planned 
sampling is that if all evaluation can be conducted at one work site, random 
sampling is preferred. If work sites are dispersed, planned sampling should 
be used. On the other hand, increased QAE travel time may be a good 
investment if the work item is important. 

e. Ideally, QAE's should be staffed to the level required to 
support the selected QA Program. In reality, the QA program used must 
accommodate the availability of QAE's. A combination of QA methods should 
be considered to get the best QA program possible with a given number of 
QAE's. 



5.2.7 Outputs Subject to Surveillance . The QAE must determine what output 
to inspect to determine the Contractor's overall performance. In many 
instances a contract requirement will generate several outputs. Evaluating 
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DATE: 


TIME: 


SOURCE OF COMPLAINT: 




Organization - 




Individual - 





NATURE OF COMPLAINT: 



CONTRACT REFERENCE: 








COMPLAINT VALIDATED: 

Date - Time - 


• 


By - 




CONTRACTOR INFORMED OF COMPLAINT: 

Date - Time - 




By - 




ACTION TAKEN BY CONTRACTOR: 









WORK RE INSPECTED: 

Date - Time - By - 

Satisfactory - 
Unsatisfactory - 



FIGURE 5-1 

Customer Complaint Record 
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a single output may provide a reliable indication of the Contractor's 
overall performance. Types of outputs subject to inspection ares 

a. Work Performed. Inspection of all work performed is the best 
way to evaluate Contractor performance. This requires that the QAE be at 
the work site during performance of the work, or be there shortly thereafter 
if results of work performed are visible ( e.g. , janitorial, grounds, etc.). 

b. Records. If the QAE is unable to be at the work site at 
required times, the inspection of records, work chits, and other documents, 
(combined with spot checks of actual work performance,) may provide a 
satisfactory indication of work performance. 

c. Management Information System (MIS) . MISs usually collect 
information over a specified period of time. This information can be 
compared to a contract standard. On the basis of this comparison, 
performance can be evaluated and the performance for the specified period 
can be accepted or rejected. 

(1) For example, the vehicle down time rate is computed every 
month. A simple comparison of the rate with the maximum acceptable rate in 
the SOW explains a great deal about the level of maintenance service 
supporting the base vehicles. 

(2) 3y way of caution, one is going to use a MIS to check the 
Contractor, make sure the MIS contains reliable data. 

5.2. S Inventory of Services . Each service requirement that is to be 
monitored must have an Inventory of Services worksheet prepared. Figure 5-2 
is a sample of an inventory worksheet. Inventory worksheets are prepared at 
the same time the QA Plans are prepared. 

a. Purpose. Inventory of Services Worksheets serve two purposes. 
First and most obvious, these worksheets provide a comprehensive listing of 
locations receiving a given service. Second, these worksheets are used to 
select locations for inspection when one of the sampling methods is used. 

b. Numbering. Worksheets list each location where a service will 
be performed. Work occurences should be grouped as daily, weekly, or 
monthly depending on how surveillance is to be conducted ( i.e. , daily, 
weekly, monthly, or other.) Each work occurence, within the group, should 
then be sequentially numbered. 

5.3 QA PLANS . The most important part of implementating a QA program is 
the development of comprehensive QA Plans. (Appendix C provides a sample of 
a QA Plan for Scheduled Solid Waste Collection.) 

a. QA Plans are documentation of how the QAE intends to monitor 
specific aspects of Contractor's performance. 

b. These plans are the basis for developing QAE schedules and 
determining required QAE staffing levels. 
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INVENTORY OF SERVICES FOR: Industrial/Administrative Solid Waste Collection 
SURVEILLANCE SCHEDULE: Weekly 



NUMBER 


LOCATION/IDENTIFICATION 


NUMBER : LOCATION/ IDENTIFICATION 


1 


BLDG 


1 


1 




2 


BLDG 


1 


1 




3 


BLDG 


2 






4 


BLDG 


3 




• • 


5 


BLDG 


4 


1 North End 


• • 

• # 


6 


BLDG 


4 


2 North End 




7 


BUDG 


4 


3 South End 




8 


BLDG 


5 






9 


PARK 


ADMIN 


[ BLDG 




1C 


PARK 


MAIN! 


1 BLDG 




11 










12 










13 










14 










15 










16 










17 










18 











FIGURE 5-2 

Inventory of Services Worksheet 
(Sample) 



c. QA Plans are developed to cover all items shown in the 
Performance Requirement Summary Table. (A single plan may cover more than 
one item if surveillance of those items is compatible.) 

d. QA Plans are monitored by the QAE throughout the life of the 
contract. Once established, they do not tend to change. 

e. Recommended formats for QA Plans are described below. (A sample 
of a QA Plan is provided in Appendix C.) 

f. Each QA Plan established should have an evaluation worksheet, or 
checklist. This worksheet, or checklist, is used to document surveillance 
results. Figure 5-3 is a sample of an Evaluation Worksheet for refuse 
collection. 

5.3.1 100 Percent Inspection . 

a. Contract Requirement. Contract requirements, identified in the 
Performance Requirements Summary Table (see Figure 4-1) , are listed along 
with the performance indicators and standards. Specific clauses in the SOW 
establishing these requirements also are listed. 
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SURVEILLANCE SCHEDULE FOR INDUSTRI AL/ ADMINS ISTRATIVE AREA SOLID WASTE COLLECTION 
CONTRACT NUMBER: • 



FOR: 22 June 1981 QAE: I. Peak 



BLDG/ 

LOCATION EMPTIED DEBRIS POSITION CONDITION _____ COMMENT 



Bldg. 

#14 


P 


P 


P 


P 






Bldg. 

#19 


? 


F 


P 


P 


Trash spilled on pavement 


when dumpster emptied 


Bldg. 

#28 


P 


P 


P 


P 






Bldg. 

#41 


F 


P 


P 


? 


Container overflowing, 


scheduled pick-up yesterday 


Bldg. 

#43 


P 


P 


P 


P 






Area 
# 3 


P 


P 


P 


F 


Container lid will not 


close 


Area 

#11 


P 


P 


P 


P 






Area 

#19 


P 


P 


P 


P 






Area 

#23 


P 


P 


P 


P 





































































































































FIGURE 5-3 

Evaluation Worksheet for Scheduled 
Industrial/Administrative Area Solid Waste Collection 
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b. Primary Method of Surveillance. The primary method of 
evaluation is 100 percent inspection. 

c. AQL. AQI/s are best stated as a percentage. 

d. Quantity of Work. Define a unit ( i . e . , single occurrence) of 
work, and determine the number of units to be performed during the 
surveillance period. Performance of each unit will be evaluated. 

e. Level of Evaluation. This is not applicable. 

f. Sample Size. This is not applicable. 

g. Sample Selection Procedure. This is not applicable. 

h. Evaluation Procedure. List any procedures or checklists used 
when doing the inspection. Evaluation procedures should be in enough detail 
to allow others to continue the same manner of inspection using the same 
evaluation criteria. 



i. Analysis of Results. The analysis of evaluation results for 100 
percent inspection is straight forward. The Observed Defect Rate (ODR) is 
computed as follows: 



ODR = 



Number of Defects 
Number Units of Work 



100 



) 



5.3.2 Planned Sampling. 



a. Contract Requirement. Contract requirements, identified in the 
Performance Requirements Summary Table (see Figure 4-1) , are listed alcng 
with the performance indicators and standards. Specific clauses in the SOW 
establishing these requirement are listed. 

b. Primary Method of Surveillance. The primary method of 
surveillance is planned sampling. Validated complaints are a good 
supportive surveillance method. 

c. AQL. AQL's should be stated as an absolute value ( e. g . 3 per 
surveillance period) . 

d. Quantity of Work. Define a unit ( i.e. single occurrence) of 
work and determine the number of units to be performed during the 
surveillance period. 

e. Level of Surveillance. If more than one level of surveillance 
is desired, it is recommended that three carefully defined levels of 
surveillance be identified and that criteria be established for switching 
from one level to another. These surveillance levels are: 

Normal - Applied to good but not exceptional contractor 

performance. This level of surveillance to be used 
when contract is first implemented. 
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Reduced - Applied in the case of exceptional contractor 
performance. 

Increased - Applied in the case of poor contractor performance. 

f. Sample Size. Determination of sample size for planned sampling 
is subjective. In order to provide consistent surveillance , the rationale 
for selecting a sample size must be identified. Sample size will vary 
depending on the level of surveillance used. 

g. Sample Selection Criteria. The criteria for sample selection 
must be documented and applied consistently from surveillance period to 
surveillance period. If there is no consistency , trends in contractor 
performance cannot be detected. 

h. Evaluation Procedure. List any procedures or checklists used 
when doing inspection. Evaluation procedures should be in enough detail to 
allow others to continue the same manner of inspection using the same 
evaluation criteria. 

i. Analysis of Results. The Observed Defect Rate (ODR) for a 
planned inspection is the total number of defects documented during the 
surveillance period. All defects detected by customer complaints and 
unschedule inspections are included in this total. The ODR is: 

ODR = number of documented defects. 

5.3.3 Random Sampling . This method, in order to be effective, must be 
properly applied. (Appendix D describes the mechanics of random sampling.) 

a. Contract Requirement. Contract requirements, identified in the 
Performance Requirements Summary Table (see Figure 4-1) , are listed along 
with the performance indicators and standards. Specific clauses in the SOW 
establishing these requirements are listed. 

b. Primary Method of Surveillance. Random sampling is the primary 
evaluation method. Validated complaints and unscheduled inspections may be 
considered as secondary methods. WARNING information collected by other 
surveillance methods car, never be combined with information gathered by 
random sampling. Evaluation results collected by other methods serve only 
as supportive data. 

c. AQL. For evaluation by random sampling, AQL*s are stated as a 
percentage and predefined as 5, 10, 15, 20 or 25 percent. Sample sizes for 
AQL's of 15, 20 and 25 percent are defined in the Sample Size Tables but are 
not generally recommended. 

d. Quantity of Work Performed. Define a unit of output for each 
service that is subject to inspection. It is important to accurately 
determine the quantity of work performed in order to select the appropriate 
sample size. 
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e. Level of Surveillance. The level of surveillance will be set at 
one of three levels, and the level of surveillance may be adjusted monthly 
depending on the Contractor's performance. 

(1) Initial level of surveillance is normal surveillance, level 

II. 

(2) Reduced Surveillance. If contractor performance has been 
■excellent, " the level of surveillance could be reduced to level I. 

(3) Increased surveillance. If, on the other hand, performance 
during the past surveillance period was poor, surveillance could be 
increased to level III. 

f. Sample Size. Sample size for random sampling is determined by 
use of tables and is a function of the AQL, quantity of work performed and 
level of surveillance. (Sample size table are provided in Appendix D.) 

g. Sampling Procedure. To assure that samples are selected 
completely at random, a random number table must be used. (This table is 
provided as part of Appendix E. } 

h. Evaluation Process. List any procedures or checklists used when 
doing the inspection. Evaluation procedures should be in enough detail to 
allow others to continue the same manner of inspection using the same 
evaluation criteria. 

i. Analysis of Results. Analysis for random sampling will consist 
of computing the Observed Defect Rate (ODR) for the past surveillance period. 

Number of Defects , 

ODR a x 100 

Number Evaluations Conducted 

The number of defects used in computing the ODR is derived from 
the samping process. Defects detected through validated complaints or 
unscheduled inspections cannot be used. 

5.3.4 Validated Customer Complaints A Validated Complaint is any customer 
complaint identifying a Contractor defect that the QAE has validated by 
documentation based on an on-site visit . Complaints not validated may not 
be used. 



a. Contract Requirement. Contract requirements, identified in the 
Performance Requirements Summary Table (see Figure 4-1) , are listed along 
with the performance indicators and standards. Specific clauses in the SOW 
establishing these requirements are listed. 

b. Primary Method of Surveillance. The primary method of 
surveillance is validated complaints. A secondary evaluation method that 
may be used is unscheduled inspections. 

c. AQL. AQL's for Validated Complaints should be stated in terms 
of number of occurrences per surveillance period. 
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d. Quantity of Work. Although the quantity of work will not have 
direct effect on validated complaint evaluations, this information is useful 
when putting results in perspective. 

e. Level of Surveillance. This is not applicable. 

f. Sample Size. Tnis is not applicable. 

g. Sample Selection Criteria. This is not applicable. 

h. Evaluation Procedure. Document how validation of complaints is 
to be performed. Evaluation procedures should be in enough detail to allow 
others to continue the same manner of inspection using the same evaluation 
criteria. 



i. Analysis of Results. Determine the number of validated 
complaints for the past surveillance period. If there is a good customer 
complaint program, changes in the number of complaints per surveillance 
period may be useful in detecting changes in the Contractor's overall level 
of performance. The Observed Defect Rate (ODR) for validated complaints is: 

ODR = number of documented defects 

5.3.5 Unscheduled Inspections . Unscheduled inspections do not have formal 
QA Plans. This type of surveillance should be used as a secondary, or 
supportive, surveillance method and as such, be subject to the QA Plan of 
the primary method. 
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CHAPTER 6 SURVEILLANCE 



6. GENERAL. The key to assuring satisfactory performance from service 

contracts is adequate Government surveillance of Contractor performance. 
Hit-or-miss surveillance by untrained personnel is an invitation to poor 
performance. The more prone (historically) a particular type of work is to 
shoddy performance, the more necessary it is to assign an adequate number of 
trained and qualified personnel (QAE ' s) who are familiar with the contract 
surveillance. The QAE is a key person in service contract management. The 
QAE serves as the eyes and ears of the SCM and as such must demonstrate a 
large degree of common sense since many facets of the job are subjective and 
open to criticism. NAVFAC Quality Assurance Evaluator (QAE) Training 
Manual, MO-326.2, provides in-depth information on QAE duties and the 
surveillance process. 

6.1 SURVEILLANCE PROCESS . The surveillance process is a system composed 
of many key elements. Figure 6-1 illustrates this system. 

6.1.1 Inputs . The key input to surveillance is contract requirements. 
These requirements dictate what work the Contractor is to perform and what 
the QAE is to evaluate. Next in importance is the Contractor's work 
schedule. The schedule is necessary in order for the QAE to know when work, 
which is not scheduled by contract requirements, is to be performed. The 
intensity of surveillance is influenced to a degree by the Contractors past 
performance. During the surveillance period the number and type of customer 
complaints received will affect the QAE's schedule. 

6.1.2 Process . The process, as displayed in Figure 6-1, may be divided 
into four main parts: planning for surveillance, conducting surveillance, 
analysis of surveillance results, and taking action based on those results. 

a. Planning for surveillance includes QA Plans and Monthly QAE 
Schedules. QA Plans are developed prior to contract award and in most cases 
remain unaltered throughout the life of the contract. Monthly QAE 
Surveillance Schedules are developed, based on QA Plans, at the start of 
each surveillance period (a period is usually one month). 

b. Contract surveillance is conducted in accordance with QA Plans 
and the QA Monthly Schedule. If, during the surveillance period, major 
discrepancies are noted (and documented) , the QAF. will alert the SCM that 
action should be taken. 

c. At the end of the surveillance period, "documented" surveillance 
results are analyzed to determine the Contractors overall level of 
performance. 

d. Based on the Contractor's performance, there are several courses 
of action that may be taken. First, deductions will be made for all 
observed and documented cases of non-compliance, regardless of the 
Contractor's overall level of performance. Other specific actions that may 
be taken include: "jaw boning" the Contractor; issuance of a Contract 
Discrepancy Report (CDR) , Cure Notice or Show Cause Notice; and contract 
termination. The person taking action may be the SCM, ROIC, OIC, or 
Commander, NAVFAC. Regardless of the course of action, GOOD DOCUMENTATION 
IS REQUIRED. 
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INPUTS 



PROCESS 




FIGURE 6-1 

Contract Surveillance Process 
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6.2 QUALITY ASSURANCE EVALUATOR (QAE) . QAE's must have qualifications 
in both the technical aspects of the contracted function and contract 
surveillance methods. 

a. Technical expertise for evaluating work quality is found within 
the organizational component that would be responsible for work 
accomplishment if it were to be performed by Government forces. The QAE 
should be drawn from this component. (In the case of CA conversions the 
activity will retain 4 percent of the organizational component being 
converted to contract performance. This 4 percent is intended for contract 
management - i.e. SCM and QAE duties.) 

b. The TRCO is the person usually responsible for selecting QAE's. 
Once selected for QAE duties, candidate QAE's must be trained in contract 
surveillance methods. This training is available through the QAE Training 
Course offered periodically by NAVFAC Engineering Field Divisions (EFD's). 

6.2.1 Authority . QAE derives his authority by delegation from the SCM and 
acts as his representative when authorized. In no case can the SCM delegate 
authority to the QAE that he himself does not have. 

a. Delegation of authority is by letter from the SCM to the QAE. 

b. The QAE has no authority to allow deviations from essential 
contract requirements, but when authorized by his superior, he may approve 
minor deviations not involving change in contract time, price, or basic 
design. 



c. The QAE has no authority to direct or interfere with the methods 
of performance by the Contractor or to issue instructions directly to any of 
the Contractor's personnel unless the methods being used are unsafe. 

6.2.2 Re spons ib i 1 i ti es . The QAE, because of his familiarity with the 

contract, the Contractor, and the customer, is involved with several aspects 
of Service Contracting. 

a. Specific QAE responsibilities will depend upon local conditions, 
size of the contract, QAE collateral duties, etc. Assigned responsibilities 
should be agreed upon by the SCM, the QAE, the TRCO, and other interested 
parties Defore the contract start date. Assigned QAE responsibilities should 
be stated in writing. 

b. Specific QAE duties are to: 

o Review plans and specifications prior to IFB/RFP; 
o Conduct Contractor pre-bid site visits; 

o Assist in pre-award surveys; 

o Review Contractor schedules and advise SCM of acceptability; 
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o Attend pre-bid and pre-start conferences? 

o Coordinate provision of Government furnished space, utili- 
ties, equipment and material; 

o Prepare Quality Assurance Plans; 

o Prepare surveillance schedules; 

o Perform surveillance and document Contractor's performance; 

o Conduct surveillance on accomplishment of re-performed work; 

o Monitor labor, safety and security practices, and document 
results; 

o Recommend payment deductions for unsatisfactory work to 
SCM; and 

o Assist in the preparation of Government estimate for change 
orders. 

6.2.3 STAFFING . An adequate level of QAE staffing is required to make any 
QA program work well. There are two approaches to staffing for contract 
surveillance: (a) write the QA program to accommodate the number of QAE's 
currently available, or (b) to write the QA program to provide the desired 
level of surveillance and staff to that program. Obviously the latter is 
the preferred approach. The chief problem is converting the desired level 
of surveillance into manhour requirements. QA Plans, and the subsequent QAE 
Schedules, provide a means of determining QAE manhour requirements. These 
documents, if properly prepared, will identify QAE staffing requirements for 
contract surveillance. 

6.3 QAE SCHEDULE . The development of an effective evaluation schedule 
should be of the utmost importance to the QAE. The evaluation schedule 
allows the QAE to plot where he should be on any given day of the week. By 
developing a balanced inspection schedule, a QAE can be much more effective 
in his job. It allows the QAE to plan his workday in advance to the best 
advantage. It also allows him to utilize his time and eliminate some 
potentially wasteful actions (for example, excessive travel time between 
inspections). QAE Schedules serve three purposes, they are: 

a. Optimizing Time. The QAE will use his established schedule to 
plan his work. By making maximum use of a good schedule, the QAE will 
optimize use of his time. 

b. Management Control. The SCM is provided a copy of each QAE 
Schedule. The SCM has the responsibility to see that surveillance of 
service contracts is properly conducted. The QAE Schedule provides him the 
information necessary to monitor the QA program. 
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c. Audit. The QAE Schedule, along with completed evaluation 
reports, provide an audit trail for contract surveillance. The established 
schedule as updated during execution, should reflect what was actually 
accomplished . 

6.3.1 Schedule Development . QAE schedules are based on established QA 
Plans. When developing monthly schedules, the QAE will use the evaluation 
worksheets developed for that month. As he prepares his schedule he may 
find it convenient to combine surveillance requirements in order to 
streamline the daily schedule. When developing schedules it may be 
necessary to modify QA Plans in order to achieve the most effective 
allocation of QAE time. However, plans based on random sampling CANNOT be 
modified in this manner. 

a. The Schedule. An example of a QAE's Schedule is shown in Figure 
6-2. This example shows only a 6-day schedule. The QAE must make up enough 
sheets to include each day of the month. Along the top of the schedule, 
insert the items to be checked during the month. Along the bottom of the 
schedule, indicate the number of observations to be made during the month. 

b. Filling In and Updating the Schedule. To fill in the inspection 
schedule, the QAE refers to the QA Plans for each service being monitored. 
The QA Plan is used to determine the inspections (observations) to be made 
during the month. 

(1) Contract surveillance must cover all hours of operation. 
Random observations are scheduled at night, on weekends, and on holidays 
when services are performed during these periods. Areas that are monitored 
on a set schedule are included in the monthly schedule. This monthly 
schedule shows where and what the QAE is monitoring at all times. 

(2) Post any changes to the schedule weekly, and send copies to 
the SCM and to the TRCO. Document and explain the reasons for each change. 
Actual surveillance activity recorded on the evaluation worksheets must be 
comparable to the monthly schedule. 

(3) As updated, one must be able to conduct a complete audit 
trail from the monthly schedule, to observing the QAE perform sampling, to 
completion of the evaluation worksheet. 

(4) After the schedule is completed and filled in, this form is 
"FOR OFFICIAL USE ONLY" and must not be shown to the Contractor. 

c. Review and Approval. The QAE* s Schedule must be submitted to 
the SCM for review and approval. The QAE is responsible for posting 
changes, as they occur, to the schedule throughout the month. This schedule 
becomes a formal part of the surveillance documentation, and, as such, it 
must be auditable. 

6.4 SURVEILLANCE . Contract surveillance involves using the QA Plan 

called for in the monthly schedule. 
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6.4,1 Performance of Surveillance . Contract surveillance is performed in 
accordance with the QA Plan: method of surveillance, sample size/selection, 

and evaluation critieria are specified in this plan. 

a. The QA program must provide an adequate and consistent level of 
surveillance. It is important that QA plan be followed and deviations 
documented in order that this program can be audited. 

b. Timing of Inspections: 

(1) In some cases inspection will have to be conducted during 
the period of work performance. For example: 

- When the Contractor is performing maintenance on a piece 
of equipment, the QAE may have to inspect the work before the equipment is 
fully reassembled. 



- If work on a building 's electrical or plumbing systems i 
performed, the QAE would have to inspect before siding or dry wall is 
replaced. 



- To determine if the Contractor is maintaining a base 
shuttle bus schedule, the QAE must be at the stop at the scheduled time. 

(2) For daily services, such as custodial services, the QAE 
should conduct inspections shortly after work performance, but prior to 
occupant use. 



(3) In many cases services performed will provide outputs of a 
lasting nature and may be inspected days after actual performance. Work, 
such as painting, resurfacing of roads, glass replacement, tree pruning, 
etc., are examples of this type of work. 

(4) Some services performed by the Contractor may be 
inspectable at any time; for example; if a grounds maintenance contract 
requires a level of maintenance (vs. frequency of work) , the QAE will be 
monitoring the condition of the grounds rather then work performed ( e >g . 
grass to be between 2 and 4 inches in height.) A watch standing requirement 
such as guard service or fire protection requires Contractor personnel on 
duty 24 hours a day. 

c. It is good practice to make surveillance findings, good or bad, 
available to the Contractor on a daily basis. Provided information does not 
relieve the Contractor of his Quality Assurance efforts but is intended to 
keep the Contractor advised of the Government's perception of the quality of 
performed work. 

6.4.2 Documentation . Just as services required of the Contractor have 
outputs. Government surveillance has outputs. One of the key outputs is 
documentation, which consists of: the QAE's monthly schedule, completed 

evaluation worksheets, records of customer complaints, and any other 
material that reflects the quality/quantity of Contractor performance. 
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SCHEDULE FOR WEEK OF : Jane 22 thru 27 



QAE: 



DAY 



TIME 


MON 


TUE 


WEN 


THU 


FRI 


SAT 


0700 


P-INSP 
MESS 
HALL #2 


P-INSP 
MESS HALL 
#4 






R-INSP 
BLDG. 1,9 
13,20,31 




0800 




Meet 

with SCM, 


MONITOR 

TRASH 

COLLECTION 


R-INSP 
BLDG. 11, 
14,18,20 


MONITOR 

TRASH 




0900 


P-INSP 

CONT. 

QC FILE 


ROIC, AND 
CONT. REP 
TO DISCUSS 


IN RES. 
AREA A 




COLLECTION 
IN RES. 
AREA B 




1000 




CHANGE 

ORDERS 




P-INSP 

WASH 

RACK 






1100 














1200 


LUNCH 


LUNCH 


LUNCH 


LUNCH 






1300 


PAPER 




PINSP 

SANITARY 




LUNCH 




1400 


WORK 

IN 

OFFICE 




LANDFILL 

OPERATIONS 




MEET 




1500 


R-INSP 
BLDG. 1 
5 f 32, 41 


R-INSP 
AREA 5,9 
11,15 






WITH 

SCM 




1600 






R-INSP 
AREAS 3,5, 
7,17,19 


P-INSP 

COMMISARY 


P-INSP 
MESS HALL 
#2 




1700 














1800 














1900 















NOTE: P-INSP = Planned Inspections 

R-INST = Random Inspections 



FIGURE 6-2 

QAE Surveillance Schedule 
(Sample) 



a. Recording Observations. Evaluation Worksheets are used for each 
QA Plan. They are used to tally information on scheduled observations and 
noted defects. Each observation in the sample is recorded on the Worksneet, 
and the Worksheets then become a formal Government record. 

b. Potential Unsatisfactory Performance. If surveillance indicates 
that the number of defects for the month may exceed the AQL, the QAE should 
try to identify and document the cause of the problem. 

(1) The cause of the problem may lie with the Government. 

(a) Is delivery of Government-provided material or 
equipment the problem? 

(b) Are Government employees (civilian or military) 
disrupting the Contractor's work efforts? 

(2) If the cause of the problem lies with the Contractor, the 
QAE should determine why. 

(a) Does the Contractor have an adequate number of people, 
or properly trained people, at the work site? 

(b) Is Contractor work supervision adequate? 

(c) Is the Contractor's QC program identifying these 
problems? (It should be.) 

(d) Are the proper equipment and materials being used? 

(e) Is the work method used capable of producing the 

required output? 

c. During the course of the contract the QAE retains a copy of all 
QAE Schedules, Evaluation Worksheets, and checklists. At the end of the 
contract period, the QAE forwards these records for inclusion in the 
contract file. However, when a specific service becomes unsatisfactory 
during a surveillance period, a copy of the inspection documentation 
supporting the contract discrepancy is forwarded to the SCM/ROIC for action. 

d. As mentioned above it is good practice to keep the Contractor 
appraised of surveillance results. One way to do this is to provide the 
Contractor's Representative a copy of the Evaluation Worksheet. It is 
recommended that Contractor's Representative initial the original Evaluation 
Worksheet indicating that he has received a copy. 

6.5 SURVEILLANCE RESULTS . It is the QAE's duty to make QA evaluation 

results known to the SCM who then is responsible for taking the appropriate 
action. At the end of the month the QAE will tally the results for all 
Evaluation Worksheets, checklists, etc., to determine the Contractor's 
overall performance with respect to each QA Plan. 
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6.5.X Excellent Performance . If the Contractor has performed in the best 
possible manner and there fcere few defects noted (Observed Defect Rate (ODR) 
less than 1/2 the AQL) , the QAE might suggest that: 

a. The Contractor should be notified by the SCM that he is 
performing satisfactorily. 

b. The level of surveillance might be reduced. 

c. Deductions will be made on all documented defects. 

6.5.2 Good Performance . When a Contractor's quality control program 
works, good performance results. If the result of a QAE's surveillance 
shows consistently good performance, the amount of surveillance can be 
decreased. 

a. Deductions will be made on all documented defects. 

b. Reduced Surveillance. Inspection can be reduced when the 
following conditions have been met for a surveillance period. 

(1) The preceding month's work (or number of months as 
specified in the QA Plan) has been acceptable. 

(2) The percentage of defects in the preceding month (s) is less 
than one half of the AQL. 

(3) The normal sample size is being used. 

(4) The TRCO and the SCM agree to use reduced inspection. 

c. Returning to Normal Surveillance. When reduced surveillance is 
in effect, return to normal inspection the next month under the following 
conditions: 



(1) When the percentage of defects exceeds the AQL under 
reduced sampling, or 

(2) The TRCO and the SCM deem it necessary to return to normal 

inspection. 

6.5.3 Questionable Performance . An outcome of questionable performance 
can only result when random sampling is the surveillance method used. 

a. Random sampling procedures take into consideration potential 
errors in results. Since random sampling only provides an estimate of the 
true defect rate, a margin for error must be used. This is done by 
specifying the accuracy desired of the ODR, as compared to the true defect 
rate. Accuracy is defined to be one half of the AQL. If, for example, the 
AQL was 10 percent, accuracy would be 5 percent. It is this gray area, 
where the ODR falls between the AQL and 1.5 times the AQL, that results are 
questionable. 

b. The recommended actions for questionable performance are: 
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(1) Deduct for all documented defects. 



(2) If there are a significant number of validated customer 
complaints and/or defects detected by unscheduled inspections, issue a 
Contract Discrepancy Report (CDR) . 

(3) Go to an increased level of evaluation. 

6,5.4 Poor Performance . If the Contractor has displayed poor performance 
( i .e . , the ODR exceeds the AQL, or for random sampling the ODR exceeds 1.5 
times the AQL) , then the following are the actions normally taken in 
addition to deductions. 

a. If the QAE first determines that the Government created any of 
the discrepancies, these are not to be counted against the Contractor's 
performance. When the Government has caused the Contractor to perform in an 
unsatisfactory manner, the QAE prepares a letter to be sent to the 
responsible organization, requesting corrective action. The QAE sends it to 
the responsible organization through the SCM. 

b. If the Government did not cause the discrepancy, the QAE tells 
the Contractor's site manager, in person, when discrepancies occur and asks 
the Contractor to correct the problem. The QAE makes a notation on the 
Evaluation Worksheet of the date and time the deficiency was discovered and 
has a Contractor's representative initial the entry on the checklist. 

c. Increased Surveillance. The level of surveillance can be 
increased when the following conditions have been met for a sampling period: 

(1) The preceding surveillance period (last month's inspection) 
has been unsatisfactory (ODR exceeds AQL) . 

(2) Normal sample size is being used. 

(3) The TRCO and the SCM agree to increased inspection. 

(4) Use the Sample Size shown in Table III, Appendix E, (or go 
to 100 percent inspection) 

(5) Return to Normal Sample Size if after one month the ODR is 
less than the AQL. 

d. When the Contractor is responsible for failing to meet the 
limits of satisfactory performance (the AQL) , the SCM issues a Contract 
Discrepancy Report (CDR) to the Contractor (discussed in 6.6.2 below). If 
the failure is serious enough, issue the CDR at the time of the 
unsatisfactory performance rather than at the end of the month. 

e. If the Contractor does not achieve satisfactory performance in 
that specific service by the end of the next month, the SCM issues another 
CDR, and the ROIC may call in the Contractor Representative for a personal 
review of the problem. 
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f. If a third CDR must be issued, the ROIC should consider issuing 
a cure notice, (However, a cure notice can be issued sooner, if necessary,) 

g. Depending on the Contractor's overall performance, the 
Government may issue a Show Cause letter if the reply to a cure notice is 
unsatisfactory, (EFD 02 approval is required,) 

6.6 TAKING ACTION . The QAE may check the Contractor’s performance and 
document any non-compliance, but only the OIC may take formal action against 
the Contractor for unsatisfactory performance. This section lists the 
normal steps to be taken by contract administration when the QAE reports 
these deficiencies. The actions listed are not hard-and-fast rules and 
represent a minimum level of action. More serious action can be taken 
sooner. 



6.6.1 Deductions . NAVFAC policy is that deductions will be made for each 
observed and documented defect. Extrapolated deductions based on random 
sampling will not be used. 

a. The QAE makes a recommendation on the amount of payment 
deductions to be made based on documented deficiencies multiplied by the 
price shown in the Schedule of Deductions, or in the Items of Bid for 
indefinite quantity work items, and the amount of liquidated damages shown 
in the "Consequences of Contractor's Failure to Perform" clause. 

b. When the Contractor's performance is unsatisfactory, i.e. , 
exceeds the AQL as defined in the QA Plan and a formal action is indicated, 
the QAE, the TRCO, the SCM, the ROIC, and the Contract Specialist meet to 
determine what action is appropriate. 

c. If a decision is reached to not take a monetary deduction, the 
reasons are to be documented. The ROIC must indicate agreement with the 
decision by signing the decision documentation. 

d. Example Deductions : 

(1) Example 1. The QAE has conducted surveillance of a 
contract requirement using planned sampling supported by validated customer 
complaints. At the end of the month results are: 

o Number of inspections conducted - 100 

o Number of defects found through planned inspection - 3 
o Number of customer complaints - 15 
o Number of complaints validated - 11 

(a) The Contractor was not given the opportunity to per- 
form the work due to time constraints. 

(b) Observed Defect Rate (ODR) is 14 (3 + 11) • Assuming 
AQL was 10 defects/month, the Contractor performance is POOR. A CDR should 
be issued. 



19.4 



(c) The established price for each unit of work is $10 
(taken from the Schedule of Deductions) . Recommended deduction is $140 plus 
an additional 10 percent for administrative costs which are allowed under 
the "CONTRACTOR'S CONSEQUENCES OF FAILURE TO PERFORM" clause of Section 
(10004, Total amount recommended for deduction is $154. 

(2) Example 2. Random sampling has been the method used to 
monitor the Contractor's performance. Results at the end of the month are: 

o Number of inspections conducted - 87 

o Number of defects found through random sampling - 6 

o Number of customer complaints - 5 

o Number of complaints validated - 2 

o Number of defects documented 

by Unscheduled Inspections - 2 

(a) The Contractor was not given the opportunity to 
perform the work due to time constraints. 

(b) The ODR is 6.9 percent (6 divided by 87). NOTE: The 

2 defects identified by customer complaints and the 2 day unscheduled 
inspections are not included in the ODR computation. 

(c) The established price for each work occurrence is $15. 
Recommended deduction is $150 ($15 X 10 defects; all defects are used for 
deduction calculations), plus $15 for administrative costs. (The 
"CONSEQUENCES OF CONTRACTOR'S FAILURE TO PERFORM" Clause, Section 00003 
allows an additional 10 percent for defective work that is either 
re-performed by the Contractor or left unperformed.) Total amount 
recommended for deduction is $165. 

(3) Example 3. The QAE has used 100 percent inspection to 
evaluate the Contractor's performance. At the end of the month results are: 

o Number of inspections conducted - 85 

o Number of defects found - 7 

o Number of defects corrected by 
Contractor - 4 

o Number of defects corrected by Government 
Employees - 3 

(a) The nature of required work was such that it could be 
re-performed, and it was necessary that all work be accomplished during the 
month . 

(b) The ODR is 7. Since this work was very important to 
the activity's mission the AQL was set at zero. The Contractor's overall 
performance is POOR. A CDR should be issued. 
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(c) The established price per unit of work is $500, The 
recommended deduction is $2000 based on the following: 

- Work re-performed by the Contractor is subject to a 
10 percent deduction for administrative costs as specified in the 
"CONSEQUENCES OF CONTRACTOR'S FAILURE TO PERFORM" clause. Section 00003. 

(In this case administrative cost is taken to mean the cost of 
re-inspection; therefore, the Government must be able to show that 
re-inspection was performed.) This amounts to $200 ($500 x 4 x 10 percent). 

- Work performed by Government employees is subject 
to 20 percent deduction to reflect the cost of the liquidated damages as 
specified in the "CONSEQUENCES OF CONTRACTOR'S FAILURE TO PERFORM" clause. 
Section 00003. If the work in question was bid lump sum, and the 
Contractor's invoice reflects that work, the cost of that work must be 
deducted. This amounts to $1800: $300 ($500 X 3 x 20 percent) plus $1500 
($500 X 3) for the cost of the work. (NOTE: If the work in question was 
bid on an indefinite quantity work item and the Contractor’s invoice did not 
reflect the work in question, only $800 would be deducted for that work 
performed by Government employees.) 

6.6.2 Contract Discrepancy Report (CDR) . CDR's are written to identify 
documented cases of poor Contractor performance. The Contractor, upon 
receiving a CDR, must identify, in writing, how future occurrences of the 
problem will be prevented ( i.e. , the corrective action he intends to take) . 
3ased on the Contractor's response, the Government may or may not take 
further action. 

a. When the Contractor's overall performance for any given contract 
requirement is classified as "poor", the QAE should recommend to the SCM 
that a CDR (Figure 6-3) be issued. 

b. The QAE is responsible for identifying the problem that caused 
the poor performance. This information is not relayed to the Contractor. 

The QAE will use this information 'to evaluate the Contractor's response. 

The Contractor should have a Quality Control Program that gives him feedback 
on his performance. 

c. If the Contractor's response is likely to correct the problem, 
the QAE should recommend to the SCM that further Government action is not 
required with the exception of an increased level of surveillance. If the 
response is not likely to correct the problem, the QAE should identify why 
it is not adequate and recommend further action to be taken by the 
Government. 
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1. CONTRACT NUMBER: 



2. TO: (Contractor & Manager’s Name) 3. FROM: (Name of SCM/ROIC) 



4, DISCREPANCY OR PROBLEM: (Describe in detail: Include reference to 

SOW Directive: Attach continuation sheet if necessary.) 



5. SIGNATURE: (SCM/ROIC) 



6. DATE: 



7. TO: (Contracting Officer) 



FROM: (Contractor) 



8. CONTRACTOR RESPONSE: (Contractor’s proposed solution to correct future 

occurrences of the problem, use Continuation Sheet if necessary) 



9. SIGNATURE: (Contractor Representative) 



DATE: 



10. GOVERNMENT EVALUATION: (Is the Contractor’s response a viable solution 

to the problem?) 



11. GOVERNMENT ACTIONS: (Does the Government accept, propose modification, 

or reject the Contractor’s response?) 



12. CLOSE OUT: 




QAE Signature: 


Date: 


SCM Signature: 


Date : 


ROIC Signature: 


Date : 



FIGURE 6-3 

Contract Discrepancy Report 
(Sample) 
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APPENDIX C 



SAMPLE QUALITY ASSURANCE (QA) PLAN 
QA PLAN FOR SCHEDULED INDUSTRIAL SOLID WASTE COLLECTION 



1. CONTRACT REQUIREMENT , Clause 3.3 of Section 00005 specifies the re- 
quirements for scheduled industrial solid waste collection. The contractor 
is required to establish collection schedules. Collection standards are: 

- Scheduled pick-ups are made within four hours of the established 
time. 

- Pick-ups are made outside of normal rush hours (or other site speci- 
fic time) . 

- Noise levels not to exceed 90 dBA as measured on a general purpose 
sound level meter, (ANSI standard SI. 4-1961). 

No more then five pieces of debris within 15 yard radius of con- 
tainer after pick-up. 

Containers re-positioned within five feet of specified location. 



2. PRIMARY METHOD OF SURVEILLANCE . Random sampling supported by customer 
complaints. 

3. ACCEPTABLE QUALITY LEVEL (AQL) . 10 percent. 

4. QUANTITY OF WORK . Appendix H to the contract specifies 85 sites for 
scheduled industrial solid waste collection. The contractor supplied' col- 
lection schedule specifies 50 of these sites for weekly collection and the 
remaining 35 sites on a twice weekly schedule. This gives a total of 480 
work occurrences per month ((50 X 4) + (35 X 4 X 2) = 480). 

5. LEVEL OF SURVEILLANCE . Normal surveillance (Level II) is recommended 
initially. If contractor performance is good ( i .e . , Observed Defect Rate 
(ODR) is 1/2 the AQL, or less) , for 2 months in a row, then reduced surveil- 
lance (Level I) should be considered. 



If the Contractors performance is questionable for the past month ( l.e . , 
ODR greater than the AQL but less than or equal to 1.5 times the AQL) and 
surveillance was used, return to normal surveillance. 



If Contractor performance is unsatisfactory in any month ( i.e. , ODR is 
greater than 1.5 times the AQL), then increased surveillance (Level III) 
should be implemented for the following month. 

6. SAMPLE SIZE . Using the Sample Size Tables, the following sample sizes 
are required for an AQL of 10 percent and a population of 480: 



LEVEL OF SURVEILLANCE NO. PER MONTH NO. PER WEEK* 



Normal (Level II) 
Reduced (Level I) 
Increased (Level III) 



109 

34 

• 178 



♦Assumes four weeks per month. 



28 

9 

45 
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7. SAMPLE SELECTION PROCEDURE . Since random sampling is the method of 
evaluation to be used, sample selection will be by a random process. The 
selection process is as follows: 

a. Collection sites are numbered from 1 to 120. 

b. On a monthly basis select four sets of random numbers of sizes in- 
dicated above ( i .e . , 28 for Level II surveillance) . Each set of numbers may 
have duplicate numbers selected for those sites on a twice weekly collection 
schedule. Numbers may be duplicated between sets. 

c. Each set of numbers selected will be matched against the numbered 
sites. These sites are the collection sites to be monitored for each re- 
spective week. 

d. Weekly evaluation worksheets are prepared for selected sites. 

8. EVALUATION PROCEDURES . The QAE will visit each selected site on the day 
collection is scheduled. Site visits, in general, should be conducted four 
hours after scheduled pick-up time. The QAE should arrange to be at the 
site at the time of pick-up if there have been problems reported with re- 
spect to debris or noise. 

9. ANALYSIS OF RESULTS . At the end of the month, the QAE is to count the 
number of pick-ups classified as unsatisfactory and compute an overall defect 
rate for the month. The Observed Defect Rate (ODR) is the number of unsatis- 
factory pick-ups divided by the sample size. 



ODR 



(Total # U's) 
(Total # U's + S's) 



X 100 



Since this QA Plan is based on random sampling, unsatisfactory pick-ups 
detected by customer complaints cannot be used in computing the ODR. Un- 
satisfactory pick-ups detected by customer complaints will be used in deter- 
mining a course of action when the contractor's observed performance is 
questionable ( i .e . , ODR greater than the AQL but less than 1.5 times the 
AQL) . 
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APPENDIX D 



RANDOM SAMPLING 



1. GENERAL , Random sampling is a surveillance method based on statistical 
theory. The key element of random sampling is that each and every occurrence 
of work has an equal chance of being evaluated. In order to achieve the 
desired end results , knowledge of the Contractor’s overall performance based 
on evaluation of only part of the work, surveillance by random sampling must 
be applied properly. Key elements of random sampling are: 

a. Sample size is specified for a given population to achieve a pre- 
determined level of statistical accuracy. 

b. The sample of work occurrences selected for evaluation must be se- 
lected by a random process in which each occurrence has equal chance for 
selection. 

c. Once an evaluation schedule has established, it must be followed 
through the surveillance period (i.e., monthly schedules). 

d. Surveillance data gathered by other methods (i.e., customer com- 
plaints and unscheduled inspections) can not be combined with data gathered 
by random sampling. 

e. Assessment of 'the Contractor's overall performance, projected from 
the observed condition of the sample, will always have the potential to be 
in error. Statements as to overall performance should be stated as "The 
Contractor’s overall defect rate is in excess of X %". (Where "X" equals 
the observed defect rate minus one half the AQL - ex. ODR (9.2%) - 1/2 AQL 
( 10 %) = 4 . 2 % . ) 

2. MECHANICS OF RANDOM SAMPLING . Random sampling is a structured approach 
based on statistics to contract surveillance. As such, there is a set pro- 
cedure in its application. The mechanics of applying random sampling are as 
follows. 

3. POPULAT ION. The total number of work occurrences for a given function 
that are to be performed during the surveillance period must be known or 
accurately estimated. The Inventory of Service Worksheet is used to deter- 
mine population size. 

a. When work is scheduled, population size is easy to determine. 

Example: Activity X has 30 dumpsters, 70 are emptied weekly and 10 

are emptied twice a week. The population, total number of work occurrences 
per month, is 360 (70x4) + (10x4x2) . 

b. When work is unscheduled, population size must be estimated. 

Example: The number of service requests for the past six months 

were: 
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Jan 321 
Feb 301 
Mar 295 
Apr 337 
May 340 
Jun 320 
Total 1914 

The average number of service calls per month has been 319 (1914/6) . This 
would be the expected population for service calls for next month unless 
there is some known reason to expect a change. 

4. SAMPLE SI2E . Sample size requirements are based on AQL, population size, 
and level of surveillance. Sample size tables are used to determine sample 
size. Tables for normal surveillance, reduced surveillance, and increased 
surveillance are attached. 

a. Select the table with the desired level of surveillance (Tables I, 

II, or III) . 

b. Select the column with the required AQL (.05, .10, .15, .20, or .25). 

c. Select the" row that is closest to the population size, preferably 
the next largest entry. 

d. The number indicated by the row and column selection is the sample 
size required for surveillance in one surveillance period. 

(1) Daily surveillance requirements will be determined by dividing 
the required period’s sample size by the number of days that surveillance is 
to be conducted. 

(2) Weekly surveillance requirements will be determined by dividing 
the required sample size by the number of weeks in a period. 

(3) When computing weekly or daily sample sizes, always round up to 
the next whole number ( e .g . 45 monthly samples required and 20 work days per 
month results in a daily sample size of 3 - (45/20) = 2.25 and rounded up 
results to 3) . 

5. Sample Selection . The final thing to be decided in sampling is how the 
sample will be drawn. The objective in the method is to insure that the 
sample is random (that is, that all services have an equal chance of being 
selected) . To achieve random selection, use a random numbers table as ex- 
plained in the following examples. (A random numbers table. Table IV, is 
attached) . Most items will fall into one of these examples. 

a. Use Of The Random Numbers Table . The random numbers in Table IV are 
arranged in groups of two. 

(1) To use the table, begin by picking at a random a group of num- 
bers on any page of the table. This is usually done by closing the eyes and 
pointing with a pencil or finger to some initial group. 
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(2) To identify additional random numbers, follow a pattern. Go 
along a given line to its end and then along the next line to its end and so 
on through the table until enough numbers have been selected or until the 
table ends. 

(3) If the table ends and there are still more numbers to select, 
go back to the beginning of the table and continue using the same pattern. 

Use various patterns alternately, for example, use lines for one sample, use 
columns for the next sample, and use a diagonal pattern for the third sample. 

b. How To Use the Random Numbers Table To Identify a Random Sample of 
Consecutively Numbered Work orders . Suppose one has to identify a random 
sample of 97 work orders for evaluation. (Sample size is based on a popula- 
tion of 319 using normal surveillance.) This can be done at the beginning 
of the month (before the work orders are written) or at the end of the month. 

(1) If there are, or might be, 319 consecutively numbered work 
orders to select from, then one begins by listing the lowest work order num- 
ber (known or projected). This could be #001, or possibly 443, or any other 
sequentially assigned number. List the highest work order number (known or 
projected); in this case, it could be #319 or 762. For this example, use 
work orders numbered #443 to 762. 

(a) Select 97 three digit numbers from Table IV using a con- 
sistent pattern. 

(b) If random numbers selected are not between 443 and 762, 
discard the number outside the designated range and select a new number. 

(2) For example, using the initial entry on Table IV we' would select 
number 441. This number is too low. The next number, going down the column, 
is 343 again too low. The third number selected is 749. This number falls 
in the range of work order number (443-762) subject to inspection. So work 
order number 749 is selected to be inspected. The next work order number 
selected is 523. 

(3) This process would be continued until three work orders are 
selected. 

c. How To Use The Random Numbers Table To Identify A Random Sample From 
A Group of Items . If a number of items need to be sampled that are not con- 
secutively numbered, the simplest solution is to list the identifiers, for 
all the items in a column, on a piece of lined paper. 

(1) Next, number the lines consecutively, beginning with the number 
one. Now use the random number table to draw the sample from the line num- 
bers. A selected line number leads to the identifier located on that line, 
and that identifier tells which item to sample. For example, if one chooses 
to sample a set of work orders with attached sales slips, one is not going 
to have to have a set of consecutively numbered work orders because not every 
work order has a sales slip attached. 
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(2) List the work orders with sales slips in a column, number each 
line in the column, and randomly select enough line numbers to make up the 
sample. 

3. How To Use The Random Numbers Table To Identify a Random Sample of 
Pays . Suppose one wants to identify four days in the month on which to sam- 
ple something. The days of the month can be numbered 01 to 31 (or less, as 
appropriate) . 

(1) It is best to use a starting point different from the one used 
in the previous example. For the purpose of this example, it is being used 
again. 

(2) One can move down the column from number to number until the 
first number between 01 and 31 is spotted. In this case, it is 22. Thus 
the 22nd day of the month is selected for sampling. 

(3) Continuing in this fashion, one discovers that 11 is the next 
number selected. This number is disregarded. Proceed in this manner until 
the four days for sampling have been identified. In our example, the 4 days 
selected would be 22, 11, 10 and 24. 

(4) If it is not desirable to sample on weekends, discard those 
days selected that happen to fall on a weekend and continue that selection 
until the proper number of days has been selected. 

e . How To Use the Random Numbers Table To Identify a Random Sample of 
Times of Day . If one wants to select random times of day to sample a ser- 
vice such as taxi or bus service, use the 24 hour clock. 

(1) If there are any constraints during each 24-hour period, take 
them into consideration. For example, suppose that base bus service opera- 
tes between 0700 and 2300. Convert these times to minutes (e.g. 0700 = 0, 
0410 - 130, 1215 = 315, etc.) Again, using Table IV and selecting three 
digit numbers and proceeding across the line from the initial number, one 
comes to 441, or 1421 hrs, as the first random time. 

(2) The next random number is 343, or 1243 hrs. The number is good 
and so one schedules an observation for 1243 hrs. 

(3) Proceed in this manner until the desired number of sample times 
have been identified. 

f. How To Insure Variety in the Use of the Random Numbers Table . The 
use of variety in the random number table ensures that detectable patterns 
do not occur. 

(1) Success in using the tables requires consistency but also vari- 
ety. The above information should ensure that the tables are properly used 
and that the sample is randomly drawn. 

g. Other Random Numbers Generating Methods . The use of a hand held 
calculator with a random number generating capability is an alternative to 
the use of random numbers table. Using this type of calculator the QAE would 
enter the minimum value and maximum value and numbers generated would always 
be within the desired range. 



50 

75 

100 

125 

150 

175 

200 

225 

250 

275 

300 

325 

350 

375 

400 

425 

450 

475 

500 

550 

600 

650 

700 

750 

800 

350 

900 

950 

1000 

1100 

1200 

1300 

1400 

1500 

1600 

1700 

1800 

1900 

2000 

2500 

3000 

3500 



TABLE D-l 

SAMPLE SIZE REQUIREMENTS FOR REDUCED SURVEILLANCE 
(SURVEILLANCE LEVEL I) 

AQL 



05 


.10 


.15 


.20 


.25 


31 


21 


16 


11 


9 


34 


25 


17 


13 


10 


44 


27 


18 


13 


10 


49 


28 


19 


14 


10 


52 


30 


20 


14 


10 


55 


30 


20 


14 


10 


57 


31 


20 


14 


11 


59 


32 


20 


14 


11 


60 


32 


21 


14 


11 


61 


32 


21 


15 


11 


63 


33 


21 


15 


11 


64 


33 


21 


15 


11 


64 


33 


21 


15 


11 


65 


33 


21 


15 


u 


66 


34 


21 


15 


11 


67 


34 


21 


15 


11 


67 


34 


21 


15 


11 


68 


34 


21 


15 


11 


68 


34 


21 


15 


11 


69 


34 


22 


15 


11 


70 


34 


22 


15 


11 


70 


35 


22 


• 15 


11 


71 


35 


22 


15 


11 


71 


35 


22 


15 


11 


72 


35 


22 


15 


11 


72 


35 


22 


15 


11 


73 


35 


22 


15 


11 


73 


35 


22 


15 


11 


73 


35 


22 


15 


11 


74 


35 


22 


15 


11 


74 


35 


22 


15 


11 


74 


36 


22 


15 


11 


75 


36 


22 


15 


11 


75 


36 


22 


15 


11 


75 


35 


22 


15 


11 


75 


36 


22 


15 


11 


76 


36 


22 


15 


11 


76 


36 


22 


15 


11 


76 


36 


22 


15 


11 


76 


36 


22 


15 


11 


77 


36 


22 


15 


11 


77 


36 


22 


15 


11 


77 


36 


22 


15 


11 


77 


36 


22 


15 


11 


78 


36 


22 


15 


11 


78 


36 


22 


15 


11 


78 


36 


22 


15 


11 


78 


36 


22 


15 


11 


78 


36 


22 


15 


11 


78 


36 


22 


15 


11 
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50 

75 

100 

125 

150 

175 

200 

225 

250 

275 

300 

325 

350 

375 

400 

425 

450 

475 

500 

550 

600 

650 

700 

750 

800 

850 

900 

950 

1000 

1100 

1200 

1300 

1400 

1500 

1600 

1700 

1800 

1900 

2000 

2500 

3000 

3500 



TABLE D-2 

SAMPLE SIZE REQUIREMENTS FOR NORMAL SURVEILLANCE 
(SURVEILLANCE LEVEL II) 

AQL 



05 


.10 


.15 


.20 


.25 


43 


37 


32 


27 


23 


60 


49 


40 


32 


27 


76 


58 


46 


37 


29 


89 


66 


' 51 


40 


31 


101 


72 


54 


42 


32 


111 


78 


57 


43 


33 


121 


82 


60 


45 


34 


129 


86 


62 


46 


35 


137 


89 


63 


47 


35 


144 


92 


65 


48 


36 


151 


95 


66 


48 


36 


157 


97 


67 


49 


37 


162 


99 


68 


49 


37 


167 


101 


69 


50 


37 


172 


103 


70 


50 


37 


176 


105 


71 


51 


38 


181 


106 


71 


51 


38 


184 


107 


72 


51 


38 


188 


109 


72 


52 


38 


195 


111 


73 


52 


38 


201 


113 


74 


52 


39 


206 


114 


75 


53 


39 


211 


116 


75 


53 


39 


215 


117 


76 


53 


•39 


219 


118 


76 


54 


39 


222 


119 


77 


54 


39 


226 


120 


77 


54 


39 


229 


121 


78 


54 


39 


231 


122 


78 


54 


39 


236 


123 


78 


55 


40 


241 


124 


79 


55 


40 


244 


125 


79 


55 


40 


248 


126 


80 


55 


40 


251 


127 


80 


55 


40 


253 


128 


80 


55 


40 


256 


128 


80 


55 


40 


258 


129 


81 


56 


40 


260 


129 


81 


56 


40 


262 


130 


81 


56 


40 


269 


131 


82 


56 


40 


274 


132 


82 


56 


41 


277 


133 


82 


56 


41 


280 


134 


83 


57 


41 


282 


134 


83 


57 


41 


284 


135 


83 


57 


41 


287 


135 


83 


57 


41 


289 


136 


83 


57 


41 


290 


136 


84 


57 


41 


291 


136 


84 


57 


41 


292 


137 


84 


57 


41 
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TABLE D-3 

SAMPLE SIZE REQUIREMENTS FOR INCREASED SURVEILLANCE 
(SURVEILLANCE LEVEL III) 



POPULATION AQL 



SIZE 


.05 


.10 


.15 


.20 


.25 


50 


47 


43 


39 


35 


31 


75 


67 


59 


52 


46 


39 


100 


86 


74 


63 


54 


45 


125 


104 


87 


72 


60 


50 


150 


121 


98 


30 


65 


53 


175 


136 


108 


86 


69 


56 


200 


151 


116 


92 


73 


58 


225 


164 


124 


97 


76 


60 


250 


177 


132 


101 


79 


62 


275 


189 


138 


105 


81 


63 


300 


201 


144 


108 


83 


64 


325 


211 


150 


111 


85 


65 


350 


222 


155 


114 


86 


66 


375 


231 


159 


116 


88 


67 


400 


241 


164 


119 


89 


68 


425 


249 


168 


121 


90 


69 


450 


258 


172 


123 


91 


69 


475 


266 


175 


124 


92 


70 


500 


273 


178 


126 


93 


70 


550 


288 


184 


129 


95 


71 


600 


301 


189 


132 


96 


72 


650 


313 


194 


134 


97 


73 


700 


324 


198 


136 


98 


73 


750 


334 


202 


138 


99 


74 


800 


344 


206 


139 


100 


74 


850 


352 


209 


141 


101 


75 


900 


361 


212 


142 


101 


75 


950 


368 


214 


143 


102 


75 


1000 


376 


217 


144 


103 


76 


1100 


389 


221 


146 


103 


76 


1200 


401 


225 


148 


104 


77 


1300 


411 


228 


149 


105 


77 


1400 


421 


231 


150 


106 


77 


1500 


429 


234 


151 


106 


77 


1600 


437 


236 


152 


107 


78 


1700 


444 


238 


153 


107 


78 


1800 


451 


240 


154 


107 


78 


1900 


457 


241 


155 


108 


78 


2000 


462 


243 


155 


108 


78 


2500 


485 


249 


158 


109 


79 


3000 


501 


253 


159 


110 


80 


3500 


513 


256 


161 


111 


80 


4000 
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APPENDIX E 

EXTRAPOLATED DEDOCTIONS, BASED ON SAMPLING TECHNIQUES 



Given: 

AQL : 6.5%; lot size : 450 units; sample : 50 units 



Ten defectives were found in the sample. 

Maximum contract payment per month: $10,000 

Payment percentage for the service (from PR 5) : x 55 

Maximum payment for acceptable service: 3500 

Unit price ($500 divided by 450 units) 31.11 

10 defectives exceed a reject number of S: 

Defective percentage in sample: 20% 

Percentage cf sample found acceptable: 805 

Credit for corrected samples: 2.2 % 

Acceptable percentage: 82.2% 

Payment for acceptable service: $411 



Figure Z. 1 Air Fcrce Deductions Under Random Sampling. 



This figure shows how deductions would normally be made 
when a contractor exceeds the AQL for any month's perfor- 
mance, This example illustrates how credits would be applied 
when work is allowed to be reperformed. Without reperfor- 
matce, the above final payment wou-d instead be $400. 
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AQL : 55?; lot size : 50 units; defectives : 5 units. 



Maximum contract payment per month; $10,000 

Maximum payment cercentage for 

specific* service; x 4% 

Maximum payment for acceptable 

service: $400 

Unit price ($400 divided by 50) $8 

5 defectives exceed a reject number of 3: 

Percentage cf let found acceptable: 905? 

Payment for acceptable performance: 3360 



Figure E.2 Air Force Deductions Based on Planned Sampling. 



This figure illustrates how deductions are made when 
planned sampling procedures are utilized. Deductions would 
not be taken if the defectives observed were less than the 
reject number of three. When the AQL for a month's perfor- 
mance is exceeded, the contractor should be notified by 
means of a discrepancy report. If successive discrepancy 
reports must be prepared, a Show Cause or Curs Notice may be 
required . 
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Ail inems are subject to 1003 inspect ion . 



Cost of required services: 35000 

local number of work units(lot size): 100 

Unit price ($50C0 divided by 100 units): 350 

Sample size: 100 

Observed nonconforming items: 5 | 

Units reworked at government option: 3 

Units r.ct creditable for payment: 3 

Total units creditable for payment: 97 

Percentage of contract price due: 973 

Cellar value due: 34850 

liquidated damaces at 103 of 

nonconforming 'ser vie e (. 10 x 6 x 350): 330 

actual amount paid: 34820 



Figure E.3 NAVFAC Payment Under 100 Percent Inspection. 



Ibis figure illustrates NAVFAC's method of deductions 
under 1003 inspection. Liquidated damages are assessed at 
103 of the value of initial observed defect rates in the 
sample. Items may be resubmitted for credit towards final 
payment. 
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Fandom sampling is baing 
All defectives are rework 
Ccst of required services 
Total work units (lot siz 
Unit price ($5000 divided 
Sample size: 

Observed nonconforming: 
Percentage nonconforming: 
AQL : 

Units reworked at governm 

Total units creditable fo 

Percentage of contract pr 

Cellar value due: 

Liquidated damaces at 10% 
nonconforming 'service ( 

Actual amount paid: 



utilized. 




ed. 




• 

• 


$5000 


e) : 


10 00 


by 1000 units) 


$5 




1 22 




11 




91 




10% 


ent option: 


11 


r payment: 


10 00 


ice due: 


100% 




35000 


of 

.10 x 11 x $5): 


$5.5 




34994.5 



Figure E.4 NAVFAC Fandom Sampling Deductions Where ACL is 
Not Exceeded. 



Under this scheme, random sampling procedures are being 
utilized. As before, liquidated damages are assessed against 
all nonconforming items that are observed. If all items are 
reworked, as in this case, full payment less any liquidated 
damages is made. 
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Random sampling procedures are being utilized. 



Cost cf required servics $5000 

Total number of work units (lot size): 1000 

Unit price ($5000 divided by 1000) $5 

Sample size: 122 

Observed nonconforming units: 11 

Threshold ( AQL) : 10®; 

Observed percentage nonconforming: 9% 

Units reworked (at Government option): 6 

Units net creditable for payment: 5 

Total units creditable for payment: 995 

Percentage cf contract price due: 99.5% 

Cellar value due: $4975 

liquidated damaces (10% cf nonconforming 

items, or .10'x IT x $5) $5.5 

Actual amount paid: $4969.5 



Figure E.5 NA7FAC Deductions when defectives are less than 
AQL and some are reworked. 



This figure shows that random sampling is the inspec- 
tion, and that not all of the units found defective were 
reworked. Thus, deductions for liquidated damages and defec- 
tive observed sample items are taken. Thzs procedure is not 
utilized by the Air Force. 
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Fandom sampling procedures ara being utilized. 



Soma defectives are reworked in the allotted time. 

Ccst of required services: $5000 

dotal number of work units (lor size) : 1000 

Onir price ($5000 divided by 1000 units): $5 

Sample size: 122 

Observed defectives nonconforming: 17 

Percentage nonconforming: 1 n% 

Threshold (AQL) : 103 

£ xtr apclar ion of defective percentage 

to obtain total acceptable units: 860 

Units reworked (at Government opricn) : 14 

Total units creditable for payment: 874 

Percentage of contract price dua : 87.43 

Cellar value due: $4390 

Liquidated camaces at 103 of the value 
of nonconforming services based on 
extrapolated percentages (.14 x 1000 x 5 x . 1):$70 

Actual amount paid: $4320 



Figure E.6 NAfFAC reductions where AQL is exceeded and 
some defectives are reworied. 



This illustration shows that when random sampling 
reveals ebserved defect rates in the sample to be greater 
than the specified AQL, liquidated damages and payment 
deductions use sample defect percentage. Units that are 
resubmitted are credited only after the extrapolation calcu- 
lations are complete. This procedure, in concert with reli- 
able statistical techniques, should be most effective in 
gaining contractor attention to defective performance. 
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Planned sampling inspection is in effect. 



Cost of required service: $5000 

Total number of work units (lot size): 10C0 

Onit price ($5000 divided by 1000): $5 

Sample size (as desired): 122 

Observed nonconforming units: 12 

Units reworked within allotted time: 6 

Total units creditable for payment: 954 

Percentage of contract price due: 99.4% 

(Deduct only for observed defects) 

Cellar value due: $4970 

Liquidated damaces at 10% of value of 

nonconforming "ser vie es: $6 

Actual amount paid: $4964 



Figure E.7 N1VFAC Deductions under planned sampling proce- 
dures. 



The figure shows the deduction procedures for planned 
sampling . 
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APPENDIX F 

INSPECTION RESOURCE ESTI3ATI0N WORKSHEETS DEVELOPED EY 

LiNTDIV 

The following three worksheets are examples of QAS 
resource estimating worksheets that have been developed and 
promulgated by the Atlantic Division, Naval Facilities 
Engineering Command. These worksheets apply for planned, 
random and 100 percent inspection methods and should result 
in fairly accurate estimates of QAE requirements. All 
monthly totals can be accumulated to obtain annual hour 
requirements, and thus provide an estimate of personnel 
ceiling points that should either be retained or requested. 
This methodology would provide future benefits in that agen- 
cies might be able to accumulate actual hours expended for 
inspection during the performance period of CA contracts, 
and thus result in a manhour estimating factor for various 
work requirements that are performed under the contract. 

Items listed in the contract requirements columns should 
correspond to work items that are listed on the performance 
requirements summaries (PRS*s). 
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LANTNAVFACENCCOMINST 11014. 4D 

QAE WORKLOAD CALCULATIONS PLANNED SAMPLING 

( 

INSTRUCTIONS: Fill in the contract requirement, quantity of work and level of 
surveillance from Performance Requirement Summary Table. 
Calculate the number of inspections. Using the estimated hours 
per inspection, calculate the number of hours of inspection 
required per month. Refer to MO-327 for suggested surveillance 
levels* 



• 

Contract Requirement 


Quantity 
of Work 


Level of 
Surveillance 


Number of 
Inspecs . 


Hrs. per 
Inspec . 


Monthly 
Hrs. of 
Inspecs. 
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LA1JTNAVFACENGC0HIJIST 11014. 4D 

( 

QAE WORKLOAD CALCULATIONS 100Z INSPECTION 

INSTRUCTIONS: Fill in the contract requirement and quantity of work froa 
Performance Requirement Summary Table. The level of 
surveillance is 100Z. The quantity of work is the number of 
inspections. Using the estimated hours per inspection, 
calculate the number of hours of inspection required per month. 



Contract Requirement 


Quantity 
of Work 


Level of 
Surveillance 


Number of 
Inspecs . 


Hrs. per 
Ins see . 


Monthly 
Hrs. of 
Inspecs. 






o 

o 












100Z 


- 










100% 












100Z 












100% 












100% 












100% 












100% 












100% 












100% 












100Z 












100% 












100% 












100% 












100% 












100% 












100% 












100% 
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LANTNAVFACENGCOMINST 11014.4D 



v 



( QAE WORKLOAD CALCULATIONS RANDOM SAMPLING 

INSTRUCTIONS: Fill in the contract requirement, quantity of work and the 
number of inspections to be performed from Performance 
Requirement Summary Table. Indicate surveillance level. Using 
the estimated hours per inspection, calculate the number of 
hours of inspection required per month. Refer to MO-327 for 
recommended surveillance levels and associated sample size 
requirements . 



Contract Requirement 


Quantity 
of Work 


Level of 
Surveillance 


Number of 
Inspecs . 


Hrs. per 
Inspec . 


Monthly 
Hrs. of 
Inspecs . 
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APPENDIX G 

VARIOUS CA CONTRACT ADMINISTRATION ORGANIZATIONS STUDIED EY 

NORTBDIV 
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•J 
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Le cend 

PWC = Public Works Officer 

A P WO = Assistant Public Works Officer 

F SC Office = Facilities Service Contract Office 

MCE = Maintenance Control Division 

SCM = Service Contract Manager 

EE'S = Planners and Estimators 

CAE's = Quality Assurance Evaluators 

This organization locates the FSC office as a 
completely centralized activity division. 



Figure G.1 Centralized Organization With SCM as Manager. 



j 
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Same as Figure J 

he FSC Office is centralized, bat loca 
he activity's Maintenance Control Divr 



Figure G.2 



Centralized Organization Located in MCD 
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